[
https://issues.apache.org/jira/browse/HBASE-24686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153100#comment-17153100
]
Sean Busbey commented on HBASE-24686:
-------------------------------------
Better logging here especially including who called close would help us chase
down a master failure case [~mchakka] has seen unsteadily in test lately for
branch-2.2. Basically the master's primary connection gets closed but the
master does not abort and then a bunch of stuff fails.
e.g. the master will log will show the normalizer failing
{code}
2020-07-03 09:57:42,005 ERROR
org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore: Failed to
normalize regions.
java.io.IOException: connection is closed
at
org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:241)
at
org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:797)
at
org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:768)
at
org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:727)
at
org.apache.hadoop.hbase.MetaTableAccessor.fullScanTables(MetaTableAccessor.java:215)
at
org.apache.hadoop.hbase.master.TableStateManager.getTablesInStates(TableStateManager.java:189)
at
org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1815)
at
org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:48)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:188)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
also if you try to use the shell or the java api to create tables that will
fail at a similar point once the master needs to call
{{MetaTableAccessor.getMetaHTable}}.
Right now the only indication of a connection closing is a line like this:
{code}
2020-07-03 09:52:38,751 INFO
org.apache.hadoop.hbase.client.ConnectionImplementation: Closing master
protocol: MasterService
{code}
a good starting point would be to update ConnectionImplementation with an INFO
message that says when we are done closing everything and a DEBUG message at
the start that includes the call stack.
> [LOG] Log improvement in Connection#close
> -----------------------------------------
>
> Key: HBASE-24686
> URL: https://issues.apache.org/jira/browse/HBASE-24686
> Project: HBase
> Issue Type: Improvement
> Components: Client, logging
> Affects Versions: 2.2.3
> Reporter: mokai
> Priority: Major
>
> We met some customers used hbase connection improperly, some threads call
> failed since the shared connection closed by one of the threads.
> It's better to print the details when connection closing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
