[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357267#comment-16357267
]
Ted Yu commented on HBASE-19954:
--------------------------------
Here is related test output with exception (the first line was added for
debugging):
{code}
2018-02-08 17:22:23,382 DEBUG [RS:0;cn012:38253]
util.ShutdownHookManager$ShutdownHookManagerV2(94): removing hook
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@6a0c707e
2018-02-08 17:22:23,382 DEBUG [M:0;cn012:41573] executor.ExecutorService(92):
Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/cn012:0,
corePoolSize=5, maxPoolSize=5
2018-02-08 17:22:23,382 DEBUG [M:0;cn012:41573] executor.ExecutorService(92):
Starting executor service name=M_LOG_REPLAY_OPS-master/cn012:0,
corePoolSize=10, maxPoolSize=10
2018-02-08 17:22:23,382 DEBUG [M:0;cn012:41573] executor.ExecutorService(92):
Starting executor service name=MASTER_TABLE_OPERATIONS-master/cn012:0,
corePoolSize=1, maxPoolSize=1
2018-02-08 17:22:23,389 ERROR [RS:0;cn012:38253]
helpers.MarkerIgnoringBase(159): ***** ABORTING region server
cn012.l42scl.hortonworks.com,38253,1518110542271: Unhandled: Failed suppression
of fs shutdown hook:
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@6a0c707e *****
java.lang.RuntimeException: Failed suppression of fs shutdown hook:
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@6a0c707e
at
org.apache.hadoop.hbase.regionserver.ShutdownHook.suppressHdfsShutdownHook(ShutdownHook.java:207)
at
org.apache.hadoop.hbase.regionserver.ShutdownHook.install(ShutdownHook.java:85)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:927)
at
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:187)
at
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:133)
at
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:171)
at java.security.AccessController.doPrivileged(Native Method)
{code}
I also added a debug log in
ShutdownHookManager#ShutdownHookManagerV2#addShutdownHook() which didn't show
up in the test output.
Since there was no check that hadoop tracks the ClientFinalizer before calling
#removeShutdownHook, the false return from hadoop resulted in the
RuntimeException to be thrown.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> ---------------------------------------------------------------------------------
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Priority: Major
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
