[
https://issues.apache.org/jira/browse/HBASE-30099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097434#comment-18097434
]
mazhengxuan commented on HBASE-30099:
-------------------------------------
Thanks for reporting this issue. I have submitted a fix in the following pull
request: https://github.com/apache/hbase/pull/8483
Root cause:
The RegionServer RPC server can start accepting requests before
preRegistrationInitialization() finishes initializing bootstrapNodeManager.
During this startup window, a GetBootstrapNodes RPC can reach
HRegionServer#getBootstrapNodes while bootstrapNodeManager is still null.
Themethod previously dereferenced the field unconditionally, resulting in the
reported NullPointerException.
Since HRegionServer#getBootstrapNodes is the shared accessor used by the
bootstrap-node RPC paths, handling the startup state in this method fixes the
race for all of its callers rather than adding separate null checks to
individual RPC handlers.
The patch makes the following changes:
1. Declares bootstrapNodeManager as volatile so that its initialization is
safely published from the RegionServer startup thread to RPC handler threads.
2. Reads bootstrapNodeManager into a local variable in
HRegionServer#getBootstrapNodes.
3. Returns an empty Iterator<ServerName> when bootstrapNodeManager has not been
initialized yet.
Returning an empty iterator represents the temporary absence of available
bootstrap nodes during RegionServer startup. It avoids exposing an internal
initialization race as an RPC failure, and allows bootstrap-node discovery to
continue or retry against other servers.
Once bootstrapNodeManager has been initialized, getBootstrapNodes continues to
return the nodes maintained by BootstrapNodeManager as before.
> HRegionServer.getBootstrapNodes() NPE when GetBootstrapNodes RPC arrives
> during startup
> ---------------------------------------------------------------------------------------
>
> Key: HBASE-30099
> URL: https://issues.apache.org/jira/browse/HBASE-30099
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 3.0.0-beta-1, 2.6.5, 2.5.14
> Reporter: Nick Dimiduk
> Assignee: mazhengxuan
> Priority: Major
> Labels: pull-request-available
>
> {{HRegionServer.getBootstrapNodes()}} throws {{NullPointerException: Cannot
> invoke "BootstrapNodeManager.getBootstrapNodes()" because
> "this.bootstrapNodeManager" is null}} when the {{GetBootstrapNodes}} RPC
> arrives after the RPC server starts accepting connections but before
> {{preRegistrationInitialization()}} has assigned the field. Once tripped,
> every subsequent {{GetBootstrapNodes}} RPC on that instance returns the same
> NPE to the caller.
> Observed in production on a branch-2.6 cluster during a period of fleet-wide
> RS restarts, when peer RegionServers all issued bootstrap-node discovery
> against each fresh RS during its startup window. Same class of startup race
> as HBASE-28088.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
