[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082888#comment-16082888 ] Hadoop QA commented on SENTRY-1630: --- Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12876669/SENTRY-1630.01-sentry-ha-redesign.patch against sentry-ha-redesign. {color:green}Overall:{color} +1 all checks pass {color:green}SUCCESS:{color} all tests passed Console output: https://builds.apache.org/job/PreCommit-SENTRY-Build/3006/console This message is automatically generated. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov >Priority: Blocker > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1630.01-sentry-ha-redesign.patch > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082667#comment-16082667 ] Hadoop QA commented on SENTRY-1630: --- Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12876648/SENTRY-1630.01-sentry-ha-redesign.patch against sentry-ha-redesign. {color:red}Overall:{color} -1 due to an error {color:red}ERROR:{color} mvn test exited 1 Console output: https://builds.apache.org/job/PreCommit-SENTRY-Build/2999/console This message is automatically generated. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov >Priority: Blocker > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1630.01-sentry-ha-redesign.patch > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082433#comment-16082433 ] Hadoop QA commented on SENTRY-1630: --- Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12876642/SENTRY-1630.01-sentry-ha-redesign.patch against sentry-ha-redesign. {color:red}Overall:{color} -1 due to 7 errors {color:red}ERROR:{color} mvn test exited 1 {color:red}ERROR:{color} Failed: org.apache.sentry.service.thrift.TestHMSFollower {color:red}ERROR:{color} Failed: org.apache.sentry.service.thrift.TestHMSFollower {color:red}ERROR:{color} Failed: org.apache.sentry.service.thrift.TestHMSFollower {color:red}ERROR:{color} Failed: org.apache.sentry.service.thrift.TestHMSFollower {color:red}ERROR:{color} Failed: org.apache.sentry.service.thrift.TestHMSFollower {color:red}ERROR:{color} Failed: org.apache.sentry.service.thrift.TestHMSFollower Console output: https://builds.apache.org/job/PreCommit-SENTRY-Build/2997/console This message is automatically generated. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov >Priority: Blocker > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1630.01-sentry-ha-redesign.patch > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081971#comment-16081971 ] Hadoop QA commented on SENTRY-1630: --- Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12876568/SENTRY-1630.01-sentry-ha-redesign.patch against sentry-ha-redesign. {color:green}Overall:{color} +1 all checks pass {color:green}SUCCESS:{color} all tests passed Console output: https://builds.apache.org/job/PreCommit-SENTRY-Build/2995/console This message is automatically generated. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov >Priority: Blocker > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1630.01-sentry-ha-redesign.patch > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081834#comment-16081834 ] Hadoop QA commented on SENTRY-1630: --- Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12876568/SENTRY-1630.01-sentry-ha-redesign.patch against sentry-ha-redesign. {color:red}Overall:{color} -1 due to 3 errors {color:red}ERROR:{color} mvn test exited 1 {color:red}ERROR:{color} Failed: org.apache.sentry.tests.e2e.dbprovider.TestDatabaseProvider {color:red}ERROR:{color} Failed: org.apache.sentry.tests.e2e.dbprovider.TestDatabaseProvider Console output: https://builds.apache.org/job/PreCommit-SENTRY-Build/2994/console This message is automatically generated. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov >Priority: Blocker > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1630.01-sentry-ha-redesign.patch > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081419#comment-16081419 ] Hadoop QA commented on SENTRY-1630: --- Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12876522/SENTRY-1630.01-sentry-ha-redesign.patch against sentry-ha-redesign. {color:red}Overall:{color} -1 due to an error {color:red}ERROR:{color} mvn test exited 1 Console output: https://builds.apache.org/job/PreCommit-SENTRY-Build/2993/console This message is automatically generated. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov >Priority: Blocker > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1630.01-sentry-ha-redesign.patch > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081406#comment-16081406 ] Hadoop QA commented on SENTRY-1630: --- Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12876515/SENTRY-1630.01-sentry-ha-redesign.patch against sentry-ha-redesign. {color:red}Overall:{color} -1 due to 3 errors {color:red}ERROR:{color} mvn test exited 1 {color:red}ERROR:{color} Failed: org.apache.sentry.tests.e2e.hdfs.TestHDFSIntegrationAdvanced {color:red}ERROR:{color} Failed: org.apache.sentry.tests.e2e.hdfs.TestHDFSIntegrationEnd2End Console output: https://builds.apache.org/job/PreCommit-SENTRY-Build/2991/console This message is automatically generated. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov >Priority: Blocker > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1630.01-sentry-ha-redesign.patch > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080056#comment-16080056 ] Hadoop QA commented on SENTRY-1630: --- Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12876376/SENTRY-1630.01-sentry-ha-redesign.patch against sentry-ha-redesign. {color:red}Overall:{color} -1 due to 3 errors {color:red}ERROR:{color} mvn test exited 1 {color:red}ERROR:{color} Failed: org.apache.sentry.tests.e2e.hdfs.TestHDFSIntegrationAdvanced {color:red}ERROR:{color} Failed: org.apache.sentry.tests.e2e.hdfs.TestHDFSIntegrationEnd2End Console output: https://builds.apache.org/job/PreCommit-SENTRY-Build/2970/console This message is automatically generated. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov >Priority: Blocker > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1630.01-sentry-ha-redesign.patch > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079740#comment-16079740 ] Hadoop QA commented on SENTRY-1630: --- Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12876331/SENTRY-1630.01-sentry-ha-redesign.patch against sentry-ha-redesign. {color:green}Overall:{color} +1 all checks pass {color:green}SUCCESS:{color} all tests passed Console output: https://builds.apache.org/job/PreCommit-SENTRY-Build/2966/console This message is automatically generated. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov >Priority: Blocker > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1630.01-sentry-ha-redesign.patch > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079719#comment-16079719 ] Hadoop QA commented on SENTRY-1630: --- Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12876331/SENTRY-1630.01-sentry-ha-redesign.patch against sentry-ha-redesign. {color:red}Overall:{color} -1 due to 2 errors {color:red}ERROR:{color} mvn test exited 1 {color:red}ERROR:{color} Failed: org.apache.sentry.tests.e2e.hdfs.TestHDFSIntegrationAdvanced Console output: https://builds.apache.org/job/PreCommit-SENTRY-Build/2965/console This message is automatically generated. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov >Priority: Blocker > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1630.01-sentry-ha-redesign.patch > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069258#comment-16069258 ] Hadoop QA commented on SENTRY-1630: --- Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12875137/SENTRY-1630.01-sentry-ha-redesign.patch against sentry-ha-redesign. {color:green}Overall:{color} +1 all checks pass {color:green}SUCCESS:{color} all tests passed Console output: https://builds.apache.org/job/PreCommit-SENTRY-Build/2944/console This message is automatically generated. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov >Priority: Blocker > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1630.01-sentry-ha-redesign.patch > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067937#comment-16067937 ] Hadoop QA commented on SENTRY-1630: --- Here are the results of testing the latest attachment https://issues.apache.org/jira/secure/attachment/12875023/SENTRY-1630.01-sentry-ha-redesign.patch against sentry-ha-redesign. {color:red}Overall:{color} -1 due to an error {color:red}ERROR:{color} failed to apply patch (exit code 1): The patch does not appear to apply with p0, p1, or p2 Console output: https://builds.apache.org/job/PreCommit-SENTRY-Build/2936/console This message is automatically generated. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Alexander Kolbasov >Priority: Blocker > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1630.01-sentry-ha-redesign.patch > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067512#comment-16067512 ] Vamsee Yarlagadda commented on SENTRY-1630: --- I was fortunate to hit this bug and was able to find the root cause of the problem. It looks like the issue is that we are using the same HiveMetaStoreClient instance to make multiple thrift calls in parallel and I imagine this access type is not supported by the API itself. {code} private class DbTask extends BaseTask { private final String dbName; DbTask(String dbName) { //Database names are case insensitive this.dbName = safeIntern(dbName.toLowerCase()); } @Override ObjectMapping doTask() throws TException { Database db = client.getDatabase(dbName); if (!dbName.equalsIgnoreCase(db.getName())) { {code} Here are the errors seen in the log to explain this: Sentry log reporting threads receiving different Database objects than requested: {code} [systest@vamsee-sentryha-1 ~]$ cat /var/log/sentry/hadoop-cmf-sentry-SENTRY_SERVER-vamsee-sentryha-1.vpc.cloudera.com.log.out.1 | grep -C 10 -e "does not match" 2017-06-28 12:04:23,039 INFO org.apache.sentry.service.thrift.HMSFollower: Using kerberos principal: sentry/vamsee-sentryha-1.vpc.cloudera@vpc.cloudera.com 2017-06-28 12:04:23,039 INFO org.apache.sentry.service.thrift.SentryKerberosContext: Logging in with new Context 2017-06-28 12:04:23,042 INFO org.apache.sentry.service.thrift.SentryKerberosContext: Sentry Ticket renewer thread started 2017-06-28 12:04:23,043 INFO hive.metastore: Trying to connect to metastore with URI thrift://vamsee-sentryha-1.vpc.cloudera.com:9083 2017-06-28 12:04:23,055 INFO hive.metastore: Opened a connection to metastore, current connections: 1 2017-06-28 12:04:23,055 INFO hive.metastore: Connected to metastore. 2017-06-28 12:04:23,055 INFO org.apache.sentry.service.thrift.HMSFollower: Secure connection established with HMS 2017-06-28 12:04:23,055 INFO org.apache.sentry.service.thrift.HMSFollower: HMSFollower of Sentry successfully connected to HMS 2017-06-28 12:04:23,078 INFO org.apache.sentry.service.thrift.HMSFollower: Before fetching hive full snapshot, Current NotificationID = CurrentNotificationEventId(eventId:3263). 2017-06-28 12:04:23,078 INFO org.apache.sentry.service.thrift.HMSFollower: Request full HMS snapshot 2017-06-28 12:04:23,122 WARN org.apache.sentry.hdfs.FullUpdateInitializer: Database name test_gr_db_2017_06_27_16_14_30 does not match test_gr_db_2017_06_27_16_03_44 2017-06-28 12:04:23,935 INFO org.apache.sentry.service.thrift.SentryKerberosContext: Sentry Ticket renewer thread finished 2017-06-28 12:04:24,087 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: Failed to execute task org.apache.thrift.TApplicationException: get_database failed: out of sequence response at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:670) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:657) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) at org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:347) at org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:210) at org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:246) -- at org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:210) at org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:246) at org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:178) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2017-06-28 12:04:24,089 INFO org.apache.sentry.service.thrift.HMSFollower: Hive full snapshot is Empty. Perhaps, HMS does not have any data 2017-06-28 12:04:24,095 INFO org.apache.sentry.service.thrift.HMSFollower: Before fetching hive full snapshot, Current NotificationID = CurrentNotificationEventId(eventId:3263). 2017-06-28 12:04:24,095 INFO org.apache.sentry.service.thrift.HMSFollower: Request full HMS snapshot 2017-06-28 12:04:24,130 WARN org.apache.sentry.hdfs.FullUpdateInitializer: Database name test_gr_db_2017_06_27_16_14_30 does not match gr_db_2017_06_27_16_32_48 2017-06-28 12:04:25,103 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: Fai
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067498#comment-16067498 ] Alexander Kolbasov commented on SENTRY-1630: [~vamsee] found the root cause of this problem. The FUllUpdateInitializer uses the same connection to HMS for multiple threads fetching information from HMS and this messes up the connection. We need to have a connection pool for Hive connections instead. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Priority: Blocker > Fix For: sentry-ha-redesign > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022957#comment-16022957 ] Na Li commented on SENTRY-1630: --- I have updated testing code TestHDFSIntegrationBase. The hive-site.xml is created before starting Sentry server, so HMSFollower will always have meta store URI, and its HiveMetaStoreClient will connect to the metastore created in the test. Note that the metastore created in the test is InternalMetastoreServer, not real metastore implementation. It may have concurrency issue and cause "out of order" when more than one requests are received at the same time. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Sergio Peña >Priority: Blocker > Fix For: sentry-ha-redesign > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021918#comment-16021918 ] Alexander Kolbasov commented on SENTRY-1630: We saw the issue on the real cluster. It is possible that HMS could have received two concurrent requests from two HMSFollower instances so we need to check how HMS handles such cases. [~kkalyan] encountered similar issue in unit tests while working on SENTRY-1669 - due to another bug we were requesting full HM update multiple times in a row and this eventually caused the problem. I don't think local metastore was involved since [~lina.li] fixes prevent local metastore from starting. Plus in the case of SENTRY-1669 tests there was only a single Sentry Server instance. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Sergio Peña >Priority: Blocker > Fix For: sentry-ha-redesign > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021906#comment-16021906 ] Sergio Peña commented on SENTRY-1630: - Interesting. I saw the error is happening only when the getFullHMSSnapshot was requesting 2 databases at the same time. The logs were showing that the HMS received those 2 requests, then it fails with the out of sequence error. Before receiving 2 requests, there were requests to one database only and that was successful all the times except when doing it with 2 databases. Maven ran this test 4 times, and the failure was caused when calling 2 {{get_database}} at the same time. Then I saw that HIVE-6893 fixed this issue on local metastores (something we run in our tests). This uses a new synchronized metastore client when connecting to local metastores. I tested this same code on CDH (where the Hive patch is backported) and it worked. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Sergio Peña >Priority: Blocker > Fix For: sentry-ha-redesign > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021876#comment-16021876 ] Alexander Kolbasov commented on SENTRY-1630: [~spena] Why do you think that this is related to the concurrency with local metastore? We saw it on real cluster deployments. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Sergio Peña >Priority: Blocker > Fix For: sentry-ha-redesign > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021815#comment-16021815 ] Sergio Peña commented on SENTRY-1630: - This issue is related to concurrency not handled very well on the local metastore. This only affects unit tests because is where we use local metastore. The issue is already solved on Hive 1.3.0 and 2.0. Found on HIVE-6893 but fixed on HIVE-10956 [~akolb] another reason to bump our hive version. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Sergio Peña >Priority: Blocker > Fix For: sentry-ha-redesign > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017383#comment-16017383 ] kalyan kumar kalvagadda commented on SENTRY-1630: - we are observing issues when HMSFollower is trying to create snapshot multiple times. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Sergio Peña >Priority: Blocker > Fix For: sentry-ha-redesign > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16016781#comment-16016781 ] Alexander Kolbasov commented on SENTRY-1630: [~kkalyan] has a way to reproduce it in unit tests > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Sergio Peña >Priority: Blocker > Fix For: sentry-ha-redesign > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995632#comment-15995632 ] kalyan kumar kalvagadda commented on SENTRY-1630: - I have not tested lately. I'm setting up a cluster today. As I recollect, this issue was seen when the services were restarted. > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Lei (Eddy) Xu >Priority: Critical > Fix For: sentry-ha-redesign > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SENTRY-1630) out of sequence error in HMSFollower
[ https://issues.apache.org/jira/browse/SENTRY-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995624#comment-15995624 ] Lei (Eddy) Xu commented on SENTRY-1630: --- Hi, [~kkalyan], [~akolb] Do you see this issue recently? > out of sequence error in HMSFollower > > > Key: SENTRY-1630 > URL: https://issues.apache.org/jira/browse/SENTRY-1630 > Project: Sentry > Issue Type: Sub-task > Components: Hdfs Plugin >Affects Versions: sentry-ha-redesign >Reporter: Hao Hao >Assignee: Lei (Eddy) Xu >Priority: Critical > Fix For: sentry-ha-redesign > > > When HMSFollower exits in an abnormal way, for all the following connection > would encounter out of sequence and SocketTimeoutException: Read timed out. > Looking at HIVE-6893, it seems to relate to leakage connection problem. > {noformat}2017-02-15 19:03:42,822 ERROR > org.apache.sentry.hdfs.FullUpdateInitializer: Task did not complete > successfully after 0 tries. Exception got: > org.apache.thrift.TApplicationException: get_database failed: out of sequence > response > 2017-02-15 19:03:42,827 ERROR org.apache.sentry.hdfs.FullUpdateInitializer: > Task did not complete successfully after 0 tries. Exception got: > MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > 2017-02-15 19:03:43,803 INFO hive.metastore: Closed a connection to > metastore, current connections: 0 > 2017-02-15 19:03:43,803 ERROR org.apache.sentry.service.thrift.HMSFollower: > Exception occurred persisting Hive full snapshot into DB > java.lang.RuntimeException: org.apache.thrift.TApplicationException: > get_database failed: out of sequence response > at > org.apache.sentry.hdfs.FullUpdateInitializer.createInitialUpdate(FullUpdateInitializer.java:324) > at > org.apache.sentry.service.thrift.HMSFollower.fetchFullUpdate(HMSFollower.java:343) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:244) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.thrift.TApplicationException: get_database failed: out > of sequence response > at > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:662) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:649) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:1213) > at > org.apache.sentry.hdfs.FullUpdateInitializer$DbTask.doTask(FullUpdateInitializer.java:256) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask$RetryStrategy.exec(FullUpdateInitializer.java:110) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:150) > at > org.apache.sentry.hdfs.FullUpdateInitializer$BaseTask.call(FullUpdateInitializer.java:79) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > 2017-02-15 19:03:43,849 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS{noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)