[
https://issues.apache.org/jira/browse/HBASE-30348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110767#comment-18110767
]
Ray Mattingly edited comment on HBASE-30348 at 9/2/26 7:26 PM:
---------------------------------------------------------------
I'm working on [a
draft|https://github.com/apache/hbase/compare/master...HubSpot:hbase:HBASE-30348-v2]
that solves this by:
* Allowing quotas against system tables, except for {{{}hbase:meta{}}},
{{{}hbase:quota{}}}, and {{{}hbase:namespace{}}}. Adding the namespace table is
what fixes the immediate 2.x deadlock – {{TableNamespaceManager}} reads it
during {{{}initClusterSchemaService{}}}, which runs before {{initQuotaManager}}
creates {{{}hbase:quota{}}}.
* Adding defense in depth for this genre of deadlock:
{{QuotaCache#ensureInitialized}} will run the initial load on an executor,
waiting on it for a configurable bounded duration rather than loading inline on
the RPC handler. A request that waits out the timeout proceeds with default,
unthrottled quota state because that's better than a deadlock and there's no
way for this code to really be certain that the master is appropriately
initialized.
Does that strategy sound reasonable?
was (Author: JIRAUSER286879):
I'm working on a draft that solves this by:
* Allowing quotas against system tables, except for {{hbase:meta}},
{{hbase:quota}}, and {{hbase:namespace}}. Adding the namespace table is what
fixes the immediate 2.x deadlock -- {{TableNamespaceManager}} reads it during
{{initClusterSchemaService}}, which runs before {{initQuotaManager}} creates
{{hbase:quota}}.
* Adding defense in depth for this genre of deadlock:
{{QuotaCache#ensureInitialized}} will run the initial load on an executor,
waiting on it for a configurable bounded duration rather than loading inline on
the RPC handler. A request that waits out the timeout proceeds with default,
unthrottled quota state because that's better than a deadlock and there's no
way for this code to really be certain that the master is appropriately
initialized.
Does that strategy sound reasonable?
> Throttles should work for system tables (excluding quotas, meta)
> ----------------------------------------------------------------
>
> Key: HBASE-30348
> URL: https://issues.apache.org/jira/browse/HBASE-30348
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 2.6.7
> Reporter: Ray Mattingly
> Assignee: Ray Mattingly
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1, 2.7.0, 3.1.0, 3.0.1, 2.6.8
>
>
> Right now throttles silently no-op for all system tables. Outside of the
> quotas table itself, of course, and perhaps the meta table I don't think this
> actually makes sense. We saw a pathologic build up of huge cells in
> incremental backups tables, for example, and it resulted in significant IO
> hotspots throughout our HBase clusters during backups and during bulk loads.
> We will fix that backups bug separately, but we would also love to have
> throttles in place on these backup system tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
