[
https://issues.apache.org/jira/browse/HBASE-24779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165919#comment-17165919
]
Josh Elser commented on HBASE-24779:
------------------------------------
{quote}To confirm this suspicion, a restart of such stuck RSes had allowed
replication to run smooth again?
{quote}
Yep.
{quote}did this use existing tooling? or something we could improve based off
of e.g. the canary tool? at a minimum sounds like a good docs addition for the
troubleshooting stuff in operator-tools or the ref guide
{quote}
Manual creation of a table and `assign`'ing it to RegionServers. We were not
able to validate if "bad" / "good" regionservers actually matched the
{{checkQuota()}} stack trace as I would've hoped.
{quote}
hbase 1-ish replication, hbase 2-ish replication, or hbase master-branch-ish
replication?
{quote}
This user was actually on 2-ish, but going off of git-log, it seems like this
logic has been here for some time. Surprising because I never ran into it
before. Make me curious how the heck it even happened and suspect of my
analysis to date.
> Improve insight into replication WAL readers hung on checkQuota
> ---------------------------------------------------------------
>
> Key: HBASE-24779
> URL: https://issues.apache.org/jira/browse/HBASE-24779
> Project: HBase
> Issue Type: Task
> Components: Replication
> Reporter: Josh Elser
> Assignee: Josh Elser
> Priority: Minor
>
> Helped a customer this past weekend who, with a large number of
> RegionServers, has some RegionServers which replicated data to a peer without
> issues while other RegionServers did not.
> The number of queue logs varied over the past 24hrs in the same manner. Some
> spikes in queued logs into 100's of logs, but other times, only 1's-10's of
> logs were queued.
> We were able to validate that there were "good" and "bad" RegionServers by
> creating a test table, assigning it to a regionserver, enabling replication
> on that table, and validating if the local puts were replicated to a peer. On
> a good RS, data was replicated immediately. On a bad RS, data was never
> replicated (at least, on the order of 10's of minutes which we waited).
> On the "bad RS", we were able to observe that the \{{wal-reader}} thread(s)
> on that RS were spending time in a Thread.sleep() in a different location
> than the other. Specifically it was sitting in the
> {{ReplicationSourceWALReader#checkQuota()}}'s sleep call, _not_ the
> {{handleEmptyWALBatch()}} method on the same class.
> My only assumption is that, somehow, these RegionServers got into a situation
> where they "allocated" memory from the quota but never freed it. Then,
> because the WAL reader thinks it has no free memory, it blocks indefinitely
> and there are no pending edits to ship and (thus) free that memory. A cursory
> glance at the code gives me a _lot_ of anxiety around places where we don't
> properly clean it up (e.g. batches that fail to ship, dropping a peer). As a
> first stab, let me add some more debugging so we can actually track this
> state properly for the operators and their sanity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
