[
https://issues.apache.org/jira/browse/HIVE-23164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17079188#comment-17079188
]
Eugene Chung edited comment on HIVE-23164 at 4/9/20, 10:59 AM:
---------------------------------------------------------------
[https://github.com/openjdk/jdk/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/classes/java/util/concurrent/ScheduledThreadPoolExecutor.java#L429-L432]
[https://github.com/openjdk/jdk/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/classes/java/util/concurrent/ThreadPoolExecutor.java#L1190-L1197]
[https://github.com/openjdk/jdk/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/classes/java/util/concurrent/Executors.java#L596-L621]
{code:java}
namePrefix = "pool-" + poolNumber.getAndIncrement() + "-thread-";
....
if (t.isDaemon())
t.setDaemon(false);
{code}
was (Author: euigeun_chung):
[https://github.com/openjdk/jdk/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/classes/java/util/concurrent/ScheduledThreadPoolExecutor.java#L429-L432]
[https://github.com/openjdk/jdk/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/classes/java/util/concurrent/ThreadPoolExecutor.java#L1190-L1197]
[https://github.com/openjdk/jdk/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/classes/java/util/concurrent/Executors.java#L596-L621]
{{ }}
{code:java}
namePrefix = "pool-" + poolNumber.getAndIncrement() + "-thread-";
....
if (t.isDaemon())
t.setDaemon(false);
{code}
{{}}
> server is not properly terminated because of non-daemon threads
> ---------------------------------------------------------------
>
> Key: HIVE-23164
> URL: https://issues.apache.org/jira/browse/HIVE-23164
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: Eugene Chung
> Assignee: Eugene Chung
> Priority: Major
> Attachments: HIVE-23164.01.patch,
> thread_dump_hiveserver2_is_not_terminated.txt
>
>
> HiveServer2 which receives the deregister command is at first preparing
> shutdown. If there's no remaining session, HiveServer2.stop() is called to
> shut down. But I found the case that the HiveServer2 JVM is not terminated
> even if HiveServer2.stop() has been called and processed. The case is always
> occurred when the local(embedded) metastore is used.
> I've attached the full thread dump describing the situation.
> [^thread_dump_hiveserver2_is_not_terminated.txt]
> In this thread dump, you can see some bunch of 'daemon' threads, NO main
> thread, and some 'non-daemon' thread(or user thread)s. As specified by
> [https://www.baeldung.com/java-daemon-thread], if there is at least one user
> thread exists, JVM does not terminate. (Note that DestroyJavaVM thread is
> non-daemon but it's special.)
>
> {code:java}
> "pool-8-thread-1" #24 prio=5 os_prio=0 tid=0x00007f52ad1fc000 nid=0x821c
> waiting on condition [0x00007f525c500000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000003cfa057c0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> {code}
> The thread above is created by ScheduledThreadPoolExecutor(int coreSize) with
> default ThreadFactory which always creates non-daemon thread. If such thread
> pool is not destroyed with ScheduledThreadPoolExecutor.shutdown() method, JVM
> cannot terminate! The only way to kill is TERM signal. If JVM receives TERM
> signal, it ignores non-daemon threads and terminates.
> So I have been digging modules which create ScheduledThreadPoolExecutor with
> non-daemon threads and I got it. As you may guess, it's the local(embedded)
> metastore. ScheduledThreadPoolExecutor is created by
> org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler#startAlwaysTaskThreads()
> and ScheduledThreadPoolExecutor.shutdown() is never called.
> Plus, I found another usage of creating such ScheduledThreadPoolExecutor and
> not calling its shutdown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
