[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265638#comment-17265638
]
Hudson commented on HBASE-25279:
Results for branch branch-2.4
[build #32 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/32/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/32/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/32/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/32/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/32/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons
> which will prevent clean shut downs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264871#comment-17264871 ] Viraj Jasani commented on HBASE-25279: -- >From daemonizing threads viewpoint, it seems only ZKWatcher is missing in >PR#2407 (in comparison to PR#2196). I think we are good here. For the actual issue of missing closure of ZKWatcher, we can focus on HBASE-25505. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264634#comment-17264634
]
Viraj Jasani commented on HBASE-25279:
--
>From historical context,
One of the critical bugs to make ZK sync() sync, was fixed in HBASE-24603 and
it just went in 2.3.0, 2.2.6 releases. It's not been that long. However, with
this fix, [~bharathv] also fixed possible deadlock during ZK event processing
by introducing Single threaded executor pool, which would process event
notifications from ZK and thus event processing is now decoupled from ZK cncx
EventThread context. While doing so, our internal utility
Threads.getNamedThreadFactory() was used, and this was always clumsy, we never
had any proper pick for using ThreadFactory, some code used to get
ThreadFactory from our internal utility, some used guava libs. I wanted to
clean this up (IIRC, it was this very Jira HBASE-24603 in which I saw
differences of opinion for using ThreadFactory during review and I decided to
clean this up in HBASE-24750). HBASE-24750 had consensus to start using guava
only and get rid of our utility, which is what was done and now we only use
guava ThreadFactoryBuilder everywhere. However, I missed daemonizing threads
which led to bug HBASE-25037 , which [~zhangduo] fixed it quite sooner and we
are good there.
With respect to current Jira, I think Bharath didn't want to daemonize thread
used in Single threaded executor pool in the first place. It happened just
because our internal utility always used daemon mode. See *if (!t.isDaemon())
\{ t.setDaemon(true); }* here:
{code:java}
public static ThreadFactory newDaemonThreadFactory(final String prefix,
final UncaughtExceptionHandler handler) {
final ThreadFactory namedFactory = getNamedThreadFactory(prefix);
return new ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
Thread t = namedFactory.newThread(r);
if (handler != null) {
t.setUncaughtExceptionHandler(handler);
} else {
t.setUncaughtExceptionHandler(LOGGING_EXCEPTION_HANDLER);
}
if (!t.isDaemon()) {
t.setDaemon(true);
}
if (t.getPriority() != Thread.NORM_PRIORITY) {
t.setPriority(Thread.NORM_PRIORITY);
}
return t;
}
};
}
{code}
This no longer exists in our code in the favour of using guava
ThreadFactoryBuilder.
[~zhangduo] Let me spend some time today and revisit HBASE-25037 to ensure if
we are not missing daemonizing any other place. If you could also cross verify
when you get time, that would be really great. WDYT?
> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons
> which will prevent clean shut downs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264587#comment-17264587 ] Lars Hofhansl commented on HBASE-25279: --- I agree the failure to close the watcher is the bug. We're pasting over the actual problem, and I bet that nobody will fix HBASE-25505 :) > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264557#comment-17264557 ] Bharath Vissapragada commented on HBASE-25279: -- > Why is the watcher not being closed? That is the bug. [~apurtell] I had the same opinion [above|https://issues.apache.org/jira/browse/HBASE-25279?focusedCommentId=17231063=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17231063]. I prefer not to mark them as daemon threads and instead fix the actual problem. Marking them as daemon threads may mask some other problem in the future. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264547#comment-17264547 ] Andrew Kyle Purtell commented on HBASE-25279: - Filed HBASE-25505 for follow up > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264545#comment-17264545 ] Andrew Kyle Purtell commented on HBASE-25279: - Ok, I've thought about it, and I'm going to do what [~zhangduo] recommends. Let me pick this to branch-2.4, close this, and open a new issue. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264543#comment-17264543 ] Andrew Kyle Purtell commented on HBASE-25279: - I was going to wait until we have a proper fix in place before moving forward with a release from branch-2.4. [~zhangduo] argues this restores old behavior. I would buy that argument but don't want this issue to get lost. What should we do here? > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264542#comment-17264542
]
Josh Elser commented on HBASE-25279:
bq. If they are daemonized if the process exits the thread can be terminated
in the middle of updating other state.
My assumption is that a watcher terminating abruptly is no different than
someone doing a {{kill -9}} on one of our processes. So, if a process abruptly
ends, there should already be something which is "fixing" that half-done state
(e.g. a procedure rollback).
That said, you want to leave branch-2.4 in the hung-shutdown state for now
rather than daemonize these and then figure out why they're not being closed
gracefully? I would've expected to get the workaround in place and then revert
the workaround when the lack-of-closure is figured out.
Sorry if I misread the intent from Bharath the first time around.
> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons
> which will prevent clean shut downs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264541#comment-17264541 ] Duo Zhang commented on HBASE-25279: --- It used to be daemon for a long time until IIRC, [~vjasani] contributed a patch to make use of guava ThreadFactoryBuilder to replace all the thread pool creation in HBase, we missed the setDaemon(true) call. This is just a restore of the old behavior. For me, I suggest we just restore the old behavior first. And if you think this a (old) bug, then let's try to fix it, in another issue. Anyway, you are the release manager, so this depends on you. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264539#comment-17264539 ] Andrew Kyle Purtell commented on HBASE-25279: - [~zhangduo] bq. I think this is a bug fix, without the patch it is impossible to do a clean shutdown. That's not true. The watchers should be closed. The threads will be terminated then. Leaving a ZK watcher unclosed is a leak. That's the real bug. Right? > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264537#comment-17264537 ] Andrew Kyle Purtell commented on HBASE-25279: - Why is the watcher not being closed? That is the bug. Making these threads daemon threads works around a problem but does not address the root cause. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264536#comment-17264536 ] Andrew Kyle Purtell commented on HBASE-25279: - [~elserj] [~bharathv] [~larsh] So what do we do here? There is a difference of opinion. I think making ZK watcher threads daemon threads are problematic. These respond to watch events and process them. If they are daemonized if the process exits the thread can be terminated in the middle of updating other state. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264535#comment-17264535 ] Josh Elser commented on HBASE-25279: bq. Don't commit it there, please. Not everyone agrees it is the right approach. Uh.. ok? What's the issue? What's your expectation in re-opening this then (if not to just change the fixVersion). I'm confused why we don't fix the bug and then you (or whomever has objections) can fix make whatever changes they like. I'm genuinely confused how this change is contentious. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264534#comment-17264534 ] Duo Zhang commented on HBASE-25279: --- I think this is a bug fix, without the patch it is impossible to do a clean shutdown. So let's cherry-pick it to branch-2.4? > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.2 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264533#comment-17264533 ] Andrew Kyle Purtell commented on HBASE-25279: - Don't commit it there, please. Not everyone agrees it is the right approach. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.2 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264532#comment-17264532
]
Josh Elser commented on HBASE-25279:
{quote}This was committed to master and branch-2, setting fix versions
accordingly
{quote}
Apologies. I guess I missed the branch-2.4.
> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons
> which will prevent clean shut downs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264529#comment-17264529 ] Andrew Kyle Purtell commented on HBASE-25279: - This was committed to master and branch-2, setting fix versions accordingly. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264528#comment-17264528 ] Andrew Kyle Purtell commented on HBASE-25279: - Why is this marked as fixed in 2.4.1? HBASE-25279 does not appear in the history of branch-2.4. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264527#comment-17264527 ] Lars Hofhansl commented on HBASE-25279: --- I just came across this as well. Currently my 2.4.1 the HMaster still hangs upon shutdown. When I annotated the thread names with the identifier of the ZKWatcher owning that pool I see it's on behalf of the ReplicationLogCleaner. Following the life-cycle of HFileLogCleaner, CleanerChore, and ScheduledChore I can't find anything obviously wrong. (If you called setConf more than once the previous ZKWatcher would not get closed, but that turned out to be not the problem.) [~apurtell] also tried to reproduce but he could not reproduce. [~elserj], [~vjasani] are you still seeing this problem? > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257246#comment-17257246
]
Hudson commented on HBASE-25279:
Results for branch branch-2
[build #143 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/143/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/143/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/143/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/143/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/143/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.4.1
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons
> which will prevent clean shut downs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257192#comment-17257192
]
Hudson commented on HBASE-25279:
Results for branch master
[build #170 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/170/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/170/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/170/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/170/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.4.1
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons
> which will prevent clean shut downs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257120#comment-17257120 ] Josh Elser commented on HBASE-25279: [~vjasani], it's been a while, but I believe I was able to reproduce this, yes. I think the ZK threads are already daemonized, so, while I agree with you long term that we should have a good understanding of ZK cnxn management inside our daemons, I think this is a good "cover our butts". [~bharathv], yeah, if we are ever shutting down without the ZK client being closed. I didn't exhaustively map that out. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231220#comment-17231220
]
Viraj Jasani commented on HBASE-25279:
--
HMaster creates two ZKWatchers:
1. Inherited from HRegionServer for tracking master, primary watcher: which is
closed properly once HRegionServer gets out of it's main loop i.e while
(!isStopped() && isHealthy())
2. For client to locate meta in ZK: which doesn't seem to be closed
{code:java}
if (clientQuorumServers != null && !clientZkObserverMode) {
// we need to take care of the ZK information synchronization
// if given client ZK are not observer nodes
ZKWatcher clientZkWatcher = new ZKWatcher(conf,
getProcessName() + ":" + rpcServices.getSocketAddress().getPort() +
"-clientZK", this,
false, true);
this.metaLocationSyncer = new MetaLocationSyncer(zooKeeper, clientZkWatcher,
this);
this.metaLocationSyncer.start();
this.masterAddressSyncer = new MasterAddressSyncer(zooKeeper,
clientZkWatcher, this);
this.masterAddressSyncer.start();
// set cluster id is a one-go effort
ZKClusterId.setClusterId(clientZkWatcher, fileSystemManager.getClusterId());
}
{code}
We can try defining clientZkWatcher as final instance variable and close with
shutdown() call soon after we delete clusterStateZNode?
{quote}In this case, these threads are tied to underlying ZK connections which
must be cleaned up. So, I don't think thats the right solution.
{quote}
On one hand, making this thread as daemon doesn't seem harmful at all, but on
the other hand, I agree that proper closures of ZKWatcher and thereby
underlying RecoverableZookeeper seem more important.
[~elserj] If you are observing an improper cleanup of this ExecutorService
(maybe as part of some unit tests), maybe we can take this opportunity to close
all ZKWatchers in master and RS (if required) as part of this Jira to make sure
we observe proper cleanup? WDYT?
> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons
> which will prevent clean shut downs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231063#comment-17231063 ] Bharath Vissapragada commented on HBASE-25279: -- > I'm not sure if we get into any troubles just marking this thread as daemon. > I would think not, it's just a smell to not be proactively closing stuff down > that we spawn I think daemon threads are useful if they are okay to be terminated abruptly and if there is not much state associated with them to be cleaned up. In this case, these threads are tied to underlying ZK connections which must be cleaned up. So, I don't think thats the right solution. Are you able to repro this at will? If so I can take a look. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231052#comment-17231052 ] Josh Elser commented on HBASE-25279: Yeah, I think you're right. I didn't go looking to see if it should've been closed down. In general, having any non-daemon threads in HBase processes is a smell (which is what caught my attention). Looks like HMaster creates an ZKWatcher but doesn't close it. Sounds like something else to follow-up on. I'm not sure if we get into any troubles just marking this thread as daemon. I would think not, it's just a smell to not be proactively closing stuff down that we spawn :) > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231007#comment-17231007 ] Bharath Vissapragada commented on HBASE-25279: -- [~elserj] Not sure I follow. Doesn't that mean someone hasn't closed the watcher? If you shutdown without closing the watcher (and its underlying connection), hows that clean? > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230984#comment-17230984 ] Josh Elser commented on HBASE-25279: [~bharathv], [~vjasani], be careful here please, gents. Marking the threads as daemon is super important. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
