Nemon Lou created HIVE-10817: -------------------------------- Summary: Blacklist For Bad MetaStore Key: HIVE-10817 URL: https://issues.apache.org/jira/browse/HIVE-10817 Project: Hive Issue Type: Improvement Components: HiveServer2, Metastore Affects Versions: 1.2.0 Reporter: Nemon Lou Assignee: Nemon Lou
During a reliability test ,when one of MetaStore 's machine power down ,HiveServer2 then never submit jobs to YARN. There are 100 JDBC clients (Beeline) running concurrently.And all the 100 JDBC clients hangs. After checking HiveServer2's thread stack,i find that most of the threads waiting to lock AbstractService while the one holding it is trying to connect to the bad MetaStore which has been power down.When the thread which hold this lock finally return SocketTimeoutException and release this lock,another thread will hold this lock and again stuck until socket time out. Adding a new blacklist mechanism finally solved this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)