[
https://issues.apache.org/jira/browse/RANGER-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caijialiang updated RANGER-4201:
--------------------------------
Description:
Issue description: HBase fails to start after enabling Ranger 2.4 plugin.
Problem analysis is required.
environment：
linux centos7.4
hbase 2.4.13
ranger 2.4
jdk1.8
hadoop 3.3.4
zookeeper 3.5.9
Symptom: After integrating Ranger with HBase, HBase Master cannot start, and
Region Server also crashes after running for a period of time. The same error
message occurs during both processes.
{code:java}
[master/gs-server-13481:16000:becomeActiveMaster] coprocessor.CoprocessorHost:
The coprocessor
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor threw
java.lang.ExceptionInInitializerError
java.lang.ExceptionInInitializerError
at
com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182)
at
com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175)
at
com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)
at com.sun.jersey.api.client.Client.init(Client.java:343)
at com.sun.jersey.api.client.Client.access$000(Client.java:119)
at com.sun.jersey.api.client.Client$1.f(Client.java:192)
at com.sun.jersey.api.client.Client$1.f(Client.java:188)
at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
at com.sun.jersey.api.client.Client.<init>(Client.java:188)
at com.sun.jersey.api.client.Client.<init>(Client.java:171)
at com.sun.jersey.api.client.Client.create(Client.java:683)
at
org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:228)
at
org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:193)
at
org.apache.ranger.plugin.util.RangerRESTClient.get(RangerRESTClient.java:473)
at
org.apache.ranger.admin.client.RangerAdminRESTClient.getRangerRolesDownloadResponse(RangerAdminRESTClient.java:1340)
at
org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1202)
at
org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:167)
at
org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:183)
at
org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:123)
at
org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:495)
at
org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:144)
at
org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:245)
at
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:1120)
at
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:160)
at
org.apache.hadoop.hbase.coprocessor.BaseEnvironment.startup(BaseEnvironment.java:69)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.checkAndLoadInstance(CoprocessorHost.java:285)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:171)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.<init>(MasterCoprocessorHost.java:155)
at
org.apache.hadoop.hbase.master.HMaster.initializeCoprocessorHost(HMaster.java:3870)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:902)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2175)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:520)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.glassfish.jersey.internal.RuntimeDelegateImpl
at
javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:130)
at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:97)
at javax.ws.rs.core.MediaType.valueOf(MediaType.java:172)
at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:65)
... 33 more
Caused by: java.lang.ClassNotFoundException:
org.glassfish.jersey.internal.RuntimeDelegateImpl
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:87)
at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:185)
at
javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:112)
... 36 more {code}
Steps to reproduce:
* 1.After installing HBase 2.x and Ranger 2.4, configure HBase by modifying
the hbase-site.xml file:
* Set hbase.coprocessor.master.classes to
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
* Set hbase.coprocessor.region.classes to
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
* Set hbase.coprocessor.regionserver.classes to
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
* Restart HBase and check the HBase Master logs to observe the exception and
see the Master crash soon after
solution：delete
/usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar
related issues
https://issues.apache.org/jira/browse/HBASE-22029
https://issues.apache.org/jira/browse/HBASE-22052
!image-2023-04-27-16-11-05-561.png!
Problem Analysis:
Conclusion: Exception caused by class loading order
Analysis process: The call chain is roughly as follows: master start {-}>
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start ...
->com.sun.jersey.api.client.Client.create
...{-}>com.sun.jersey.core.spi.factory.MessageBodyFactory.init
->com.sun.jersey.core.header.MediaTypes ->javax.ws.rs.core.MediaType.valueOf ->
javax.ws.rs.ext.RuntimeDelegate.getInstance ->
javax.ws.rs.ext.RuntimeDelegate.findDelegate
I have fully investigated this call chain and found that the root cause of the
problem is that javax.ws.rs.ext.RuntimeDelegate in the
javax.ws.rs-api-2.1.1.jar package in HBase did not find
org.glassfish.jersey.internal.RuntimeDelegateImpl when calling findDelegate,
which caused the error. Specifically, javax.ws.rs.ext.RuntimeDelegate first
looks for the default JAXRS_RUNTIME_DELEGATE, which is
"javax.ws.rs.ext.RuntimeDelegate". If it cannot be found, it will fallback and
load JAXRS_DEFAULT_RUNTIME_DELEGATE, which is
org.glassfish.jersey.internal.RuntimeDelegateImpl.
The method "findDelegate" in "javax.ws.rs.ext.RuntimeDelegate" is as follows:
{code:java}
public abstract class RuntimeDelegate {
public static final String JAXRS_RUNTIME_DELEGATE_PROPERTY =
"javax.ws.rs.ext.RuntimeDelegate";
private static final String JAXRS_DEFAULT_RUNTIME_DELEGATE =
"org.glassfish.jersey.internal.RuntimeDelegateImpl";
private static final Object RD_LOCK = new Object();
private static ReflectPermission suppressAccessChecksPermission = new
ReflectPermission("suppressAccessChecks");
private static volatile RuntimeDelegate cachedDelegate; protected
RuntimeDelegate() {
}
public static RuntimeDelegate getInstance() {
// Double-check idiom for lazy initialization of fields.
// Local variable is used to limit the number of more expensive
accesses to a volatile field.
RuntimeDelegate result = cachedDelegate;
if (result == null) { // First check (no locking)
synchronized (RD_LOCK) {
result = cachedDelegate;
if (result == null) { // Second check (with locking)
cachedDelegate = result = findDelegate();
}
}
}
return result;
} private static RuntimeDelegate findDelegate() {
try {
Object delegate = FactoryFinder.find(
JAXRS_RUNTIME_DELEGATE_PROPERTY,
JAXRS_DEFAULT_RUNTIME_DELEGATE,
RuntimeDelegate.class);
if (!(delegate instanceof RuntimeDelegate)) {
Class pClass = RuntimeDelegate.class;
String classnameAsResource = pClass.getName().replace('.', '/')
+ ".class";
ClassLoader loader = pClass.getClassLoader();
if (loader == null) {
loader = ClassLoader.getSystemClassLoader();
}
URL targetTypeURL = loader.getResource(classnameAsResource);
throw new LinkageError("ClassCastException: attempting to cast"
+
delegate.getClass().getClassLoader().getResource(classnameAsResource)
+ " to " + targetTypeURL);
}
return (RuntimeDelegate) delegate;
} catch (Exception ex) {
throw new RuntimeException(ex);
}
}
{code}
When the "jersey-core-1.19.3.jar" file is removed from
"/usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/", HBase can
start normally.
It is not clear which class, either "javax.ws.rs.ext.RuntimeDelegate" or
"org.glassfish.jersey.internal.RuntimeDelegateImpl", is loaded by HBase at this
point.
To determine which class is loaded and from which JAR file it is loaded, we can
first remove "/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar" and start HBase
normally. Then, we can use the Arthas tool
([https://github.com/alibaba/arthas]) to sniff and determine which class is
loaded and from which JAR file it is loaded by the HBase JVM.
After starting HBase normally, run the Arthas startup script "sudo -u hbase -EH
./as.sh " attach to the HBase process. Then, execute "sc -d
javax.ws.rs.ext.RuntimeDelegat" to check whether
"javax.ws.rs.ext.RuntimeDelegate" is loaded.
result：
{code:java}
[arthas@1719]$ sc -d javax.ws.rs.ext.RuntimeDelegate
class-info com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
code-source
/usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/jersey-core-1.19.jar
name com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name AbstractRuntimeDelegate
modifier abstract,public
annotation
interfaces
super-class +-javax.ws.rs.ext.RuntimeDelegate
+-java.lang.Object
class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
+-sun.misc.Launcher$ExtClassLoader@436a4e4b
classLoaderHash 4fca772d class-info
com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
code-source /usr/bigtop/3.2.0/usr/lib/hadoop/lib/jersey-server-1.19.jar
name com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name RuntimeDelegateImpl
modifier public
annotation
interfaces
super-class +-com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
+-javax.ws.rs.ext.RuntimeDelegate
+-java.lang.Object
class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
+-sun.misc.Launcher$ExtClassLoader@436a4e4b
classLoaderHash 4fca772d class-info javax.ws.rs.ext.RuntimeDelegate
code-source /usr/bigtop/3.2.0/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar
name javax.ws.rs.ext.RuntimeDelegate
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name RuntimeDelegate
modifier abstract,public
annotation
interfaces
super-class +-java.lang.Object
class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
+-sun.misc.Launcher$ExtClassLoader@436a4e4b
classLoaderHash 4fca772dAffect(row-cnt:3) cost in 112 ms. {code}
By removing the jersey-core-1.19.3.jar file from
/usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/, we can see that
the abstract class javax.ws.rs.ext.RuntimeDelegate is loaded by the
AppClassLoader from the javax.ws.rs-api-2.1.1.jar file located in
/usr/bigtop/3.2.0/usr/lib/hbase/lib/.
At the same time, the implementation of javax.ws.rs.ext.RuntimeDelegate, which
is com.sun.jersey.server.impl.provider.RuntimeDelegateImpl, is loaded from the
jersey-server-1.19.jar file located in /usr/bigtop/3.2.0/usr/lib/hadoop/lib/.
Note that the hadoop lib directory is appended to HBase's classpath when HBase
is started, which is why it can be loaded.
Additionally, the abstract class
com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate, which inherits from
RuntimeDelegate, is loaded from the jersey-core-1.19.jar file located in
/usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/.
However, even after removing the jersey-core-1.19.3.jar file, we can still see
that HBase loads the same version of jersey-core-1.19.jar from the hadoop lib
directory. Therefore, the root cause of the error is not a class not found due
to a version conflict, but rather due to the order in which the class loader
loads the classes.
To verify this conclusion, the simplest way is to:
# Before deleting the jersey-core-1.19.3.jar file from
/ranger-hbase-plugin-impl/, print out the loading information of all the
classes involved in the stack trace of the heap dump exception in the log.
# After deleting the jersey-core-1.19.3.jar file, print out the loading
information of all the classes involved in the stack trace of the heap dump
exception in the log.
# Use regular expressions to batch delete the timestamps from the log, and
then compare the two log files to see any differences in the class loading
information.
In the Ranger 2.4 source code, Ranger implements its own class loader,
RangerPluginClassLoader, which inherits from URLClassLoader. During
initialization, it is passed the path to the
ranger-\{componentname}{-}plugin-impl directory, and all the classes in this
directory will be loaded by RangerPluginClassLoader. The parent is set to null,
which in my understanding, is to block delegation loading. This is because all
the classes in the ranger{-}{componentname}-plugin-impl directory are
dependencies of the Ranger plugin. For example, if the version of a jar that it
depends on is lower, if the delegation loading mechanism is used, it is easy to
load the higher version of the class under the component/lib directory, leading
to class conflict issues. Therefore, parent is set to null.
Internally, RangerPluginClassLoader first tries to load the jar files under the
ranger-\{componentname}-plugin-impl directory using the parent URLClassLoader.
If it can't be loaded, it uses the componentClassLoader, which is the current
thread context's class loader, to load it.
!image-2023-04-27-16-13-44-504.png!
Next, you can take a look at the source code of the
"RangerAuthorizationCoprocessor" class in Ranger. This class initializes an
instance of "rangerPluginClassLoader" and activates it. Within this class,
there are pre and post hooks for various HBase operations, primarily for
performing range permission-related operations. This triggers the loading of
Ranger-related classes by the rangerPluginClassLoader.
!image-2023-04-27-16-14-10-803.png!
Why does the hook activate the Ranger class loader at the beginning and
de-activate it after the hook method ends? Doesn't this approach impact
performance?
!image-2023-04-27-16-14-37-202.png!
The reason for de-activating the Ranger class loader after the hook method ends
is because the rangerPluginClassLoader code sets itself as the current thread's
context class loader. If it's not deactivated, it can cause conflicts with the
lower version packages loaded in the .../ranger-hbase-plugin-impl/ directory,
which can cause issues during subsequent operations. Therefore, the conclusion
is that this involves the rangerPluginClassLoader and the AppClassLoader of the
thread's context class loader.
We also need to understand how to load an instance of "RuntimeDelegate" and how
it searches for the implementation of "javax.ws.rs.ext.RuntimeDelegate". We
should also investigate why it cannot find the implementation class and instead
looks for "org.glassfish.jersey.internal.RuntimeDelegateImpl", causing an error.
First, the "RuntimeDelegate" class is loaded via the
"javax.ws.rs.ext.RuntimeDelegate.findDelegate" method, which calls
"FactoryFinder.find" to load the class. Once loaded, the instance must be of
type "RuntimeDelegate".
{code:java}
private static RuntimeDelegate findDelegate() {
try {
Object delegate = FactoryFinder.find("javax.ws.rs.ext.RuntimeDelegate",
"org.glassfish.jersey.internal.RuntimeDelegateImpl", RuntimeDelegate.class);
if (!(delegate instanceof RuntimeDelegate)) {
Class<RuntimeDelegate> pClass = RuntimeDelegate.class;
String classnameAsResource = pClass.getName().replace('.', '/') +
".class";
ClassLoader loader = pClass.getClassLoader();
if (loader == null)
loader = ClassLoader.getSystemClassLoader();
URL targetTypeURL = loader.getResource(classnameAsResource);
throw new LinkageError("ClassCastException: attempting to cast" +
delegate
.getClass().getClassLoader().getResource(classnameAsResource) + "
to " + targetTypeURL);
}
return (RuntimeDelegate)delegate;
} catch (Exception ex) {
throw new RuntimeException(ex);
}
} {code}
The process of loading the class via "FactoryFinder.find" is contained in the
"javax.ws.rs.ext.FactoryFinder.find" method, as shown in the following code:
{code:java}
/**
* Finds the implementation {@code Class} for the given factory name,
* or if that fails, finds the {@code Class} for the given fallback
* class name and create its instance. The arguments supplied MUST be
* used in order. If using the first argument is successful, the second
* one will not be used.
* <p>
* This method is package private so that this code can be shared.
*
* @param factoryId the name of the factory to find, which is
* a system property.
* @param fallbackClassName the implementation class name, which is
* to be used only if nothing else.
* is found; {@code null} to indicate that
* there is no fallback class name.
* @param service service to be found.
* @param <T> type of the service to be found.
* @return the instance of the specified service; may not be {@code null}.
* @throws ClassNotFoundException if the given class could not be found
* or could not be instantiated.
*/
static <T> Object find(final String factoryId, final String
fallbackClassName, Class<T> service) throws ClassNotFoundException {
ClassLoader classLoader = getContextClassLoader(); try {
Iterator<T> iterator = ServiceLoader.load(service,
FactoryFinder.getContextClassLoader()).iterator();
if(iterator.hasNext()) {
return iterator.next();
}
} catch (Exception | ServiceConfigurationError ex) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId +
".", ex);
} try {
Iterator<T> iterator = ServiceLoader.load(service,
FactoryFinder.class.getClassLoader()).iterator();
if(iterator.hasNext()) {
return iterator.next();
}
} catch (Exception | ServiceConfigurationError ex) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId +
".", ex);
} // try to read from $java.home/lib/jaxrs.properties
FileInputStream inputStream = null;
String configFile = null;
try {
String javah = System.getProperty("java.home");
configFile = javah + File.separator + "lib" + File.separator +
"jaxrs.properties";
File f = new File(configFile);
if (f.exists()) {
Properties props = new Properties();
inputStream = new FileInputStream(f);
props.load(inputStream);
String factoryClassName = props.getProperty(factoryId);
return newInstance(factoryClassName, classLoader);
}
} catch (Exception ex) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId
+ " from $java.home/lib/jaxrs.properties", ex);
} finally {
if (inputStream != null) {
try {
inputStream.close();
} catch (IOException ex) {
LOGGER.log(Level.FINER, String.format("Error closing %s
file.", configFile), ex);
}
}
} // Use the system property
try {
String systemProp = System.getProperty(factoryId);
if (systemProp != null) {
return newInstance(systemProp, classLoader);
}
} catch (SecurityException se) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId
+ " from a system property", se);
} if (fallbackClassName == null) {
throw new ClassNotFoundException(
"Provider for " + factoryId + " cannot be found", null);
} return newInstance(fallbackClassName, classLoader);
}
} {code}
To summarize the "FactoryFinder.find" method, it uses the thread context class
loader to load "javax.ws.rs.ext.RuntimeDelegate". If that fails, it attempts to
load it from system properties or alternative class names. If it still cannot
be found, it tries to load "org.glassfish.jersey.internal.RuntimeDelegateImpl".
If it cannot be loaded, it throws an exception.
When the
/usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar
file was not deleted, the class loading order was as follows:
javax.ws.rs.ext.RuntimeDelegate(appClassLoader) ->
javax.ws.rs.ext.RuntimeDelegate.HeaderDelegate(RangerPluginClassLoader) ->
javax.ws.rs.ext.RuntimeDelegate.HeaderDelegateProvider(RangerPluginClassLoader
jersey-core-1.19.3.jar ranger-hbase-plugin-impl/jersey-core-1.19.3.jar).
The RangerPluginClassLoader searched for the
com.sun.jersey.spi.HeaderDelegateProvider implementation class in the
META-INF/services/com.sun.jersey.spi.HeaderDelegateProvider file of the
jersey-core-1.19.3.jar file, found the
com.sun.jersey.core.impl.provider.header.LocaleProvider class in
ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, and returned it.
However, javax.ws.rs.ext.RuntimeDelegate.findDelegate would fail because
delegate instanceof RuntimeDelegate returned false.
As a fallback, it searched for
org.glassfish.jersey.internal.RuntimeDelegateImpl, which was not found in the
hbase classpath and caused an exception to be thrown.
After removing ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, HBase can be
started normally. The class loading order is as follows:
javax.ws.rs.ext.RuntimeDelegate is loaded from
/usr/lib/hadoop/lib/javax.ws.rs-api-2.1.1.jar. Then, the implementation class
of javax.ws.rs.ext.RuntimeDelegate is searched for, which is found in
/usr/lib/hadoop/lib/jersey-server-1.19.jar->META-INF/services/javax.ws.rs.ext.RuntimeDelegate->com.sun.jersey.server.impl.provider.RuntimeDelegateImpl,
and successfully loaded.
The above class loading order was determined by modifying the code in the
RangerPluginClassLoader to print out the process of loading all classes.
!image-2023-04-27-16-15-44-539.png!
The following is a log that includes the class loading order. By analyzing this
log in conjunction with the Ranger class loader code, the above results can be
determined.
[^hbase-hbase-master.err.log]
[^hbase-hbase-master.log]
was:
Issue description: HBase fails to start after enabling Ranger 2.4 plugin.
Problem analysis is required.
environment：
linux centos7.4
hbase 2.4.13
ranger 2.4
jdk1.8
hadoop 3.3.4
zookeeper 3.5.9
Symptom: After integrating Ranger with HBase, HBase Master cannot start, and
Region Server also crashes after running for a period of time. The same error
message occurs during both processes.
{code:java}
[master/gs-server-13481:16000:becomeActiveMaster] coprocessor.CoprocessorHost:
The coprocessor
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor threw
java.lang.ExceptionInInitializerError
java.lang.ExceptionInInitializerError
at
com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182)
at
com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175)
at
com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)
at com.sun.jersey.api.client.Client.init(Client.java:343)
at com.sun.jersey.api.client.Client.access$000(Client.java:119)
at com.sun.jersey.api.client.Client$1.f(Client.java:192)
at com.sun.jersey.api.client.Client$1.f(Client.java:188)
at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
at com.sun.jersey.api.client.Client.<init>(Client.java:188)
at com.sun.jersey.api.client.Client.<init>(Client.java:171)
at com.sun.jersey.api.client.Client.create(Client.java:683)
at
org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:228)
at
org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:193)
at
org.apache.ranger.plugin.util.RangerRESTClient.get(RangerRESTClient.java:473)
at
org.apache.ranger.admin.client.RangerAdminRESTClient.getRangerRolesDownloadResponse(RangerAdminRESTClient.java:1340)
at
org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1202)
at
org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:167)
at
org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:183)
at
org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:123)
at
org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:495)
at
org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:144)
at
org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:245)
at
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:1120)
at
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:160)
at
org.apache.hadoop.hbase.coprocessor.BaseEnvironment.startup(BaseEnvironment.java:69)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.checkAndLoadInstance(CoprocessorHost.java:285)
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:171)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.<init>(MasterCoprocessorHost.java:155)
at
org.apache.hadoop.hbase.master.HMaster.initializeCoprocessorHost(HMaster.java:3870)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:902)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2175)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:520)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.glassfish.jersey.internal.RuntimeDelegateImpl
at
javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:130)
at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:97)
at javax.ws.rs.core.MediaType.valueOf(MediaType.java:172)
at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:65)
... 33 more
Caused by: java.lang.ClassNotFoundException:
org.glassfish.jersey.internal.RuntimeDelegateImpl
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:87)
at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:185)
at
javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:112)
... 36 more {code}
Steps to reproduce:
* 1.After installing HBase 2.x and Ranger 2.4, configure HBase by modifying
the hbase-site.xml file:
* Set hbase.coprocessor.master.classes to
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
* Set hbase.coprocessor.region.classes to
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
* Set hbase.coprocessor.regionserver.classes to
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
* Restart HBase and check the HBase Master logs to observe the exception and
see the Master crash soon after
solution：delete
/usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar
related issues
https://issues.apache.org/jira/browse/HBASE-22029
https://issues.apache.org/jira/browse/HBASE-22052
!image-2023-04-27-16-11-05-561.png!
Problem Analysis:
Conclusion: Exception caused by class loading order
Analysis process: The call chain is roughly as follows: master start ->
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start ...
->com.sun.jersey.api.client.Client.create
...->com.sun.jersey.core.spi.factory.MessageBodyFactory.init
->com.sun.jersey.core.header.MediaTypes ->javax.ws.rs.core.MediaType.valueOf ->
javax.ws.rs.ext.RuntimeDelegate.getInstance ->
javax.ws.rs.ext.RuntimeDelegate.findDelegate
I have fully investigated this call chain and found that the root cause of the
problem is that javax.ws.rs.ext.RuntimeDelegate in the
javax.ws.rs-api-2.1.1.jar package in HBase did not find
org.glassfish.jersey.internal.RuntimeDelegateImpl when calling findDelegate,
which caused the error. Specifically, javax.ws.rs.ext.RuntimeDelegate first
looks for the default JAXRS_RUNTIME_DELEGATE, which is
"javax.ws.rs.ext.RuntimeDelegate". If it cannot be found, it will fallback and
load JAXRS_DEFAULT_RUNTIME_DELEGATE, which is
org.glassfish.jersey.internal.RuntimeDelegateImpl.
The method "findDelegate" in "javax.ws.rs.ext.RuntimeDelegate" is as follows:
{code:java}
public abstract class RuntimeDelegate {
public static final String JAXRS_RUNTIME_DELEGATE_PROPERTY =
"javax.ws.rs.ext.RuntimeDelegate";
private static final String JAXRS_DEFAULT_RUNTIME_DELEGATE =
"org.glassfish.jersey.internal.RuntimeDelegateImpl";
private static final Object RD_LOCK = new Object();
private static ReflectPermission suppressAccessChecksPermission = new
ReflectPermission("suppressAccessChecks");
private static volatile RuntimeDelegate cachedDelegate; protected
RuntimeDelegate() {
}
public static RuntimeDelegate getInstance() {
// Double-check idiom for lazy initialization of fields.
// Local variable is used to limit the number of more expensive
accesses to a volatile field.
RuntimeDelegate result = cachedDelegate;
if (result == null) { // First check (no locking)
synchronized (RD_LOCK) {
result = cachedDelegate;
if (result == null) { // Second check (with locking)
cachedDelegate = result = findDelegate();
}
}
}
return result;
} private static RuntimeDelegate findDelegate() {
try {
Object delegate = FactoryFinder.find(
JAXRS_RUNTIME_DELEGATE_PROPERTY,
JAXRS_DEFAULT_RUNTIME_DELEGATE,
RuntimeDelegate.class);
if (!(delegate instanceof RuntimeDelegate)) {
Class pClass = RuntimeDelegate.class;
String classnameAsResource = pClass.getName().replace('.', '/')
+ ".class";
ClassLoader loader = pClass.getClassLoader();
if (loader == null) {
loader = ClassLoader.getSystemClassLoader();
}
URL targetTypeURL = loader.getResource(classnameAsResource);
throw new LinkageError("ClassCastException: attempting to cast"
+
delegate.getClass().getClassLoader().getResource(classnameAsResource)
+ " to " + targetTypeURL);
}
return (RuntimeDelegate) delegate;
} catch (Exception ex) {
throw new RuntimeException(ex);
}
}
{code}
When the "jersey-core-1.19.3.jar" file is removed from
"/usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/", HBase can
start normally.
It is not clear which class, either "javax.ws.rs.ext.RuntimeDelegate" or
"org.glassfish.jersey.internal.RuntimeDelegateImpl", is loaded by HBase at this
point.
To determine which class is loaded and from which JAR file it is loaded, we can
first remove "/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar" and start HBase
normally. Then, we can use the Arthas tool (https://github.com/alibaba/arthas)
to sniff and determine which class is loaded and from which JAR file it is
loaded by the HBase JVM.
After starting HBase normally, run the Arthas startup script "sudo -u hbase -EH
./as.sh " attach to the HBase process. Then, execute "sc -d
javax.ws.rs.ext.RuntimeDelegat" to check whether
"javax.ws.rs.ext.RuntimeDelegate" is loaded.
result：
{code:java}
[arthas@1719]$ sc -d javax.ws.rs.ext.RuntimeDelegate
class-info com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
code-source
/usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/jersey-core-1.19.jar
name com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name AbstractRuntimeDelegate
modifier abstract,public
annotation
interfaces
super-class +-javax.ws.rs.ext.RuntimeDelegate
+-java.lang.Object
class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
+-sun.misc.Launcher$ExtClassLoader@436a4e4b
classLoaderHash 4fca772d class-info
com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
code-source /usr/bigtop/3.2.0/usr/lib/hadoop/lib/jersey-server-1.19.jar
name com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name RuntimeDelegateImpl
modifier public
annotation
interfaces
super-class +-com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
+-javax.ws.rs.ext.RuntimeDelegate
+-java.lang.Object
class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
+-sun.misc.Launcher$ExtClassLoader@436a4e4b
classLoaderHash 4fca772d class-info javax.ws.rs.ext.RuntimeDelegate
code-source /usr/bigtop/3.2.0/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar
name javax.ws.rs.ext.RuntimeDelegate
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name RuntimeDelegate
modifier abstract,public
annotation
interfaces
super-class +-java.lang.Object
class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
+-sun.misc.Launcher$ExtClassLoader@436a4e4b
classLoaderHash 4fca772dAffect(row-cnt:3) cost in 112 ms. {code}
By removing the jersey-core-1.19.3.jar file from
/usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/, we can see that
the abstract class javax.ws.rs.ext.RuntimeDelegate is loaded by the
AppClassLoader from the javax.ws.rs-api-2.1.1.jar file located in
/usr/bigtop/3.2.0/usr/lib/hbase/lib/.
At the same time, the implementation of javax.ws.rs.ext.RuntimeDelegate, which
is com.sun.jersey.server.impl.provider.RuntimeDelegateImpl, is loaded from the
jersey-server-1.19.jar file located in /usr/bigtop/3.2.0/usr/lib/hadoop/lib/.
Note that the hadoop lib directory is appended to HBase's classpath when HBase
is started, which is why it can be loaded.
Additionally, the abstract class
com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate, which inherits from
RuntimeDelegate, is loaded from the jersey-core-1.19.jar file located in
/usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/.
However, even after removing the jersey-core-1.19.3.jar file, we can still see
that HBase loads the same version of jersey-core-1.19.jar from the hadoop lib
directory. Therefore, the root cause of the error is not a class not found due
to a version conflict, but rather due to the order in which the class loader
loads the classes.
To verify this conclusion, the simplest way is to:
# Before deleting the jersey-core-1.19.3.jar file from
/ranger-hbase-plugin-impl/, print out the loading information of all the
classes involved in the stack trace of the heap dump exception in the log.
# After deleting the jersey-core-1.19.3.jar file, print out the loading
information of all the classes involved in the stack trace of the heap dump
exception in the log.
# Use regular expressions to batch delete the timestamps from the log, and
then compare the two log files to see any differences in the class loading
information.
In the Ranger 2.4 source code, Ranger implements its own class loader,
RangerPluginClassLoader, which inherits from URLClassLoader. During
initialization, it is passed the path to the
ranger-\{componentname}-plugin-impl directory, and all the classes in this
directory will be loaded by RangerPluginClassLoader. The parent is set to null,
which in my understanding, is to block delegation loading. This is because all
the classes in the ranger-\{componentname}-plugin-impl directory are
dependencies of the Ranger plugin. For example, if the version of a jar that it
depends on is lower, if the delegation loading mechanism is used, it is easy to
load the higher version of the class under the component/lib directory, leading
to class conflict issues. Therefore, parent is set to null.
Internally, RangerPluginClassLoader first tries to load the jar files under the
ranger-\{componentname}-plugin-impl directory using the parent URLClassLoader.
If it can't be loaded, it uses the componentClassLoader, which is the current
thread context's class loader, to load it.
!image-2023-04-27-16-13-44-504.png!
Next, you can take a look at the source code of the
"RangerAuthorizationCoprocessor" class in Ranger. This class initializes an
instance of "rangerPluginClassLoader" and activates it. Within this class,
there are pre and post hooks for various HBase operations, primarily for
performing range permission-related operations. This triggers the loading of
Ranger-related classes by the rangerPluginClassLoader.
!image-2023-04-27-16-14-10-803.png!
Why does the hook activate the Ranger class loader at the beginning and
de-activate it after the hook method ends? Doesn't this approach impact
performance?
!image-2023-04-27-16-14-37-202.png!
The reason for de-activating the Ranger class loader after the hook method ends
is because the rangerPluginClassLoader code sets itself as the current thread's
context class loader. If it's not deactivated, it can cause conflicts with the
lower version packages loaded in the .../ranger-hbase-plugin-impl/ directory,
which can cause issues during subsequent operations. Therefore, the conclusion
is that this involves the rangerPluginClassLoader and the AppClassLoader of the
thread's context class loader.
We also need to understand how to load an instance of "RuntimeDelegate" and how
it searches for the implementation of "javax.ws.rs.ext.RuntimeDelegate". We
should also investigate why it cannot find the implementation class and instead
looks for "org.glassfish.jersey.internal.RuntimeDelegateImpl", causing an error.
First, the "RuntimeDelegate" class is loaded via the
"javax.ws.rs.ext.RuntimeDelegate.findDelegate" method, which calls
"FactoryFinder.find" to load the class. Once loaded, the instance must be of
type "RuntimeDelegate".
{code:java}
private static RuntimeDelegate findDelegate() {
try {
Object delegate = FactoryFinder.find("javax.ws.rs.ext.RuntimeDelegate",
"org.glassfish.jersey.internal.RuntimeDelegateImpl", RuntimeDelegate.class);
if (!(delegate instanceof RuntimeDelegate)) {
Class<RuntimeDelegate> pClass = RuntimeDelegate.class;
String classnameAsResource = pClass.getName().replace('.', '/') +
".class";
ClassLoader loader = pClass.getClassLoader();
if (loader == null)
loader = ClassLoader.getSystemClassLoader();
URL targetTypeURL = loader.getResource(classnameAsResource);
throw new LinkageError("ClassCastException: attempting to cast" +
delegate
.getClass().getClassLoader().getResource(classnameAsResource) + "
to " + targetTypeURL);
}
return (RuntimeDelegate)delegate;
} catch (Exception ex) {
throw new RuntimeException(ex);
}
} {code}
The process of loading the class via "FactoryFinder.find" is contained in the
"javax.ws.rs.ext.FactoryFinder.find" method, as shown in the following code:
{code:java}
/**
* Finds the implementation {@code Class} for the given factory name,
* or if that fails, finds the {@code Class} for the given fallback
* class name and create its instance. The arguments supplied MUST be
* used in order. If using the first argument is successful, the second
* one will not be used.
* <p>
* This method is package private so that this code can be shared.
*
* @param factoryId the name of the factory to find, which is
* a system property.
* @param fallbackClassName the implementation class name, which is
* to be used only if nothing else.
* is found; {@code null} to indicate that
* there is no fallback class name.
* @param service service to be found.
* @param <T> type of the service to be found.
* @return the instance of the specified service; may not be {@code null}.
* @throws ClassNotFoundException if the given class could not be found
* or could not be instantiated.
*/
static <T> Object find(final String factoryId, final String
fallbackClassName, Class<T> service) throws ClassNotFoundException {
ClassLoader classLoader = getContextClassLoader(); try {
Iterator<T> iterator = ServiceLoader.load(service,
FactoryFinder.getContextClassLoader()).iterator();
if(iterator.hasNext()) {
return iterator.next();
}
} catch (Exception | ServiceConfigurationError ex) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId +
".", ex);
} try {
Iterator<T> iterator = ServiceLoader.load(service,
FactoryFinder.class.getClassLoader()).iterator();
if(iterator.hasNext()) {
return iterator.next();
}
} catch (Exception | ServiceConfigurationError ex) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId +
".", ex);
} // try to read from $java.home/lib/jaxrs.properties
FileInputStream inputStream = null;
String configFile = null;
try {
String javah = System.getProperty("java.home");
configFile = javah + File.separator + "lib" + File.separator +
"jaxrs.properties";
File f = new File(configFile);
if (f.exists()) {
Properties props = new Properties();
inputStream = new FileInputStream(f);
props.load(inputStream);
String factoryClassName = props.getProperty(factoryId);
return newInstance(factoryClassName, classLoader);
}
} catch (Exception ex) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId
+ " from $java.home/lib/jaxrs.properties", ex);
} finally {
if (inputStream != null) {
try {
inputStream.close();
} catch (IOException ex) {
LOGGER.log(Level.FINER, String.format("Error closing %s
file.", configFile), ex);
}
}
} // Use the system property
try {
String systemProp = System.getProperty(factoryId);
if (systemProp != null) {
return newInstance(systemProp, classLoader);
}
} catch (SecurityException se) {
LOGGER.log(Level.FINER, "Failed to load service " + factoryId
+ " from a system property", se);
} if (fallbackClassName == null) {
throw new ClassNotFoundException(
"Provider for " + factoryId + " cannot be found", null);
} return newInstance(fallbackClassName, classLoader);
}
} {code}
To summarize the "FactoryFinder.find" method, it uses the thread context class
loader to load "javax.ws.rs.ext.RuntimeDelegate". If that fails, it attempts to
load it from system properties or alternative class names. If it still cannot
be found, it tries to load "org.glassfish.jersey.internal.RuntimeDelegateImpl".
If it cannot be loaded, it throws an exception.
When the
/usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar
file was not deleted, the class loading order was as follows:
javax.ws.rs.ext.RuntimeDelegate(appClassLoader) ->
javax.ws.rs.ext.RuntimeDelegate.HeaderDelegate(RangerPluginClassLoader) ->
javax.ws.rs.ext.RuntimeDelegate.HeaderDelegateProvider(RangerPluginClassLoader
jersey-core-1.19.3.jar ranger-hbase-plugin-impl/jersey-core-1.19.3.jar).
The RangerPluginClassLoader searched for the
com.sun.jersey.spi.HeaderDelegateProvider implementation class in the
META-INF/services/com.sun.jersey.spi.HeaderDelegateProvider file of the
jersey-core-1.19.3.jar file, found the
com.sun.jersey.core.impl.provider.header.LocaleProvider class in
ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, and returned it.
However, javax.ws.rs.ext.RuntimeDelegate.findDelegate would fail because
delegate instanceof RuntimeDelegate returned false.
As a fallback, it searched for
org.glassfish.jersey.internal.RuntimeDelegateImpl, which was not found in the
hbase classpath and caused an exception to be thrown.
After removing ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, HBase can be
started normally. The class loading order is as follows:
javax.ws.rs.ext.RuntimeDelegate is loaded from
/usr/lib/hadoop/lib/javax.ws.rs-api-2.1.1.jar. Then, the implementation class
of javax.ws.rs.ext.RuntimeDelegate is searched for, which is found in
/usr/lib/hadoop/lib/jersey-server-1.19.jar->META-INF/services/javax.ws.rs.ext.RuntimeDelegate->com.sun.jersey.server.impl.provider.RuntimeDelegateImpl,
and successfully loaded.
The above class loading order was determined by modifying the code in the
RangerPluginClassLoader to print out the process of loading all classes.
!image-2023-04-27-16-15-44-539.png!
The following is a log that includes the class loading order. By analyzing this
log in conjunction with the Ranger class loader code, the above results can be
determined.
> Hbase master can't start due to ranger-hbase-plugin jersey jar class loading
> order
> -----------------------------------------------------------------------------------
>
> Key: RANGER-4201
> URL: https://issues.apache.org/jira/browse/RANGER-4201
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
> Affects Versions: 2.3.0, 2.4.0
> Reporter: caijialiang
> Assignee: caijialiang
> Priority: Major
> Attachments: hbase-hbase-master.err.log, hbase-hbase-master.log,
> image-2023-04-27-16-11-05-561.png, image-2023-04-27-16-13-44-504.png,
> image-2023-04-27-16-14-10-803.png, image-2023-04-27-16-14-37-202.png,
> image-2023-04-27-16-15-44-539.png
>
>
>
> Issue description: HBase fails to start after enabling Ranger 2.4 plugin.
> Problem analysis is required.
> environment：
> linux centos7.4
> hbase 2.4.13
> ranger 2.4
> jdk1.8
> hadoop 3.3.4
> zookeeper 3.5.9
> Symptom: After integrating Ranger with HBase, HBase Master cannot start, and
> Region Server also crashes after running for a period of time. The same error
> message occurs during both processes.
>
> {code:java}
> [master/gs-server-13481:16000:becomeActiveMaster]
> coprocessor.CoprocessorHost: The coprocessor
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor threw
> java.lang.ExceptionInInitializerError
> java.lang.ExceptionInInitializerError
> at
> com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182)
> at
> com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175)
> at
> com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)
> at com.sun.jersey.api.client.Client.init(Client.java:343)
> at com.sun.jersey.api.client.Client.access$000(Client.java:119)
> at com.sun.jersey.api.client.Client$1.f(Client.java:192)
> at com.sun.jersey.api.client.Client$1.f(Client.java:188)
> at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
> at com.sun.jersey.api.client.Client.<init>(Client.java:188)
> at com.sun.jersey.api.client.Client.<init>(Client.java:171)
> at com.sun.jersey.api.client.Client.create(Client.java:683)
> at
> org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:228)
> at
> org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:193)
> at
> org.apache.ranger.plugin.util.RangerRESTClient.get(RangerRESTClient.java:473)
> at
> org.apache.ranger.admin.client.RangerAdminRESTClient.getRangerRolesDownloadResponse(RangerAdminRESTClient.java:1340)
> at
> org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1202)
> at
> org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:167)
> at
> org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:183)
> at
> org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:123)
> at
> org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:495)
> at
> org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:144)
> at
> org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:245)
> at
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:1120)
> at
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:160)
> at
> org.apache.hadoop.hbase.coprocessor.BaseEnvironment.startup(BaseEnvironment.java:69)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.checkAndLoadInstance(CoprocessorHost.java:285)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:171)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.<init>(MasterCoprocessorHost.java:155)
> at
> org.apache.hadoop.hbase.master.HMaster.initializeCoprocessorHost(HMaster.java:3870)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:902)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2175)
> at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:520)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.glassfish.jersey.internal.RuntimeDelegateImpl
> at
> javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:130)
> at
> javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:97)
> at javax.ws.rs.core.MediaType.valueOf(MediaType.java:172)
> at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:65)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException:
> org.glassfish.jersey.internal.RuntimeDelegateImpl
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:87)
> at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:185)
> at
> javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:112)
> ... 36 more {code}
> Steps to reproduce:
>
> * 1.After installing HBase 2.x and Ranger 2.4, configure HBase by modifying
> the hbase-site.xml file:
> * Set hbase.coprocessor.master.classes to
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
> * Set hbase.coprocessor.region.classes to
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
> * Set hbase.coprocessor.regionserver.classes to
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
> * Restart HBase and check the HBase Master logs to observe the exception and
> see the Master crash soon after
> solution：delete
> /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar
>
>
> related issues
> https://issues.apache.org/jira/browse/HBASE-22029
> https://issues.apache.org/jira/browse/HBASE-22052
> !image-2023-04-27-16-11-05-561.png!
> Problem Analysis:
> Conclusion: Exception caused by class loading order
>
> Analysis process: The call chain is roughly as follows: master start {-}>
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start
> ... ->com.sun.jersey.api.client.Client.create
> ...{-}>com.sun.jersey.core.spi.factory.MessageBodyFactory.init
> ->com.sun.jersey.core.header.MediaTypes ->javax.ws.rs.core.MediaType.valueOf
> -> javax.ws.rs.ext.RuntimeDelegate.getInstance ->
> javax.ws.rs.ext.RuntimeDelegate.findDelegate
> I have fully investigated this call chain and found that the root cause of
> the problem is that javax.ws.rs.ext.RuntimeDelegate in the
> javax.ws.rs-api-2.1.1.jar package in HBase did not find
> org.glassfish.jersey.internal.RuntimeDelegateImpl when calling findDelegate,
> which caused the error. Specifically, javax.ws.rs.ext.RuntimeDelegate first
> looks for the default JAXRS_RUNTIME_DELEGATE, which is
> "javax.ws.rs.ext.RuntimeDelegate". If it cannot be found, it will fallback
> and load JAXRS_DEFAULT_RUNTIME_DELEGATE, which is
> org.glassfish.jersey.internal.RuntimeDelegateImpl.
> The method "findDelegate" in "javax.ws.rs.ext.RuntimeDelegate" is as follows:
> {code:java}
> public abstract class RuntimeDelegate {
> public static final String JAXRS_RUNTIME_DELEGATE_PROPERTY =
> "javax.ws.rs.ext.RuntimeDelegate";
> private static final String JAXRS_DEFAULT_RUNTIME_DELEGATE =
> "org.glassfish.jersey.internal.RuntimeDelegateImpl";
> private static final Object RD_LOCK = new Object();
> private static ReflectPermission suppressAccessChecksPermission = new
> ReflectPermission("suppressAccessChecks");
> private static volatile RuntimeDelegate cachedDelegate; protected
> RuntimeDelegate() {
> }
> public static RuntimeDelegate getInstance() {
> // Double-check idiom for lazy initialization of fields.
> // Local variable is used to limit the number of more expensive
> accesses to a volatile field.
> RuntimeDelegate result = cachedDelegate;
> if (result == null) { // First check (no locking)
> synchronized (RD_LOCK) {
> result = cachedDelegate;
> if (result == null) { // Second check (with locking)
> cachedDelegate = result = findDelegate();
> }
> }
> }
> return result;
> } private static RuntimeDelegate findDelegate() {
> try {
> Object delegate = FactoryFinder.find(
> JAXRS_RUNTIME_DELEGATE_PROPERTY,
> JAXRS_DEFAULT_RUNTIME_DELEGATE,
> RuntimeDelegate.class);
> if (!(delegate instanceof RuntimeDelegate)) {
> Class pClass = RuntimeDelegate.class;
> String classnameAsResource = pClass.getName().replace('.',
> '/') + ".class";
> ClassLoader loader = pClass.getClassLoader();
> if (loader == null) {
> loader = ClassLoader.getSystemClassLoader();
> }
> URL targetTypeURL = loader.getResource(classnameAsResource);
> throw new LinkageError("ClassCastException: attempting to
> cast"
> +
> delegate.getClass().getClassLoader().getResource(classnameAsResource)
> + " to " + targetTypeURL);
> }
> return (RuntimeDelegate) delegate;
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> }
> {code}
> When the "jersey-core-1.19.3.jar" file is removed from
> "/usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/", HBase can
> start normally.
>
> It is not clear which class, either "javax.ws.rs.ext.RuntimeDelegate" or
> "org.glassfish.jersey.internal.RuntimeDelegateImpl", is loaded by HBase at
> this point.
>
> To determine which class is loaded and from which JAR file it is loaded, we
> can first remove "/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar" and start
> HBase normally. Then, we can use the Arthas tool
> ([https://github.com/alibaba/arthas]) to sniff and determine which class is
> loaded and from which JAR file it is loaded by the HBase JVM.
> After starting HBase normally, run the Arthas startup script "sudo -u hbase
> -EH ./as.sh " attach to the HBase process. Then, execute "sc -d
> javax.ws.rs.ext.RuntimeDelegat" to check whether
> "javax.ws.rs.ext.RuntimeDelegate" is loaded.
>
> result：
> {code:java}
> [arthas@1719]$ sc -d javax.ws.rs.ext.RuntimeDelegate
> class-info com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
> code-source
> /usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/jersey-core-1.19.jar
> name com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
> isInterface false
> isAnnotation false
> isEnum false
> isAnonymousClass false
> isArray false
> isLocalClass false
> isMemberClass false
> isPrimitive false
> isSynthetic false
> simple-name AbstractRuntimeDelegate
> modifier abstract,public
> annotation
> interfaces
> super-class +-javax.ws.rs.ext.RuntimeDelegate
> +-java.lang.Object
> class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
> +-sun.misc.Launcher$ExtClassLoader@436a4e4b
> classLoaderHash 4fca772d class-info
> com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
> code-source /usr/bigtop/3.2.0/usr/lib/hadoop/lib/jersey-server-1.19.jar
> name com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
> isInterface false
> isAnnotation false
> isEnum false
> isAnonymousClass false
> isArray false
> isLocalClass false
> isMemberClass false
> isPrimitive false
> isSynthetic false
> simple-name RuntimeDelegateImpl
> modifier public
> annotation
> interfaces
> super-class +-com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
> +-javax.ws.rs.ext.RuntimeDelegate
> +-java.lang.Object
> class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
> +-sun.misc.Launcher$ExtClassLoader@436a4e4b
> classLoaderHash 4fca772d class-info javax.ws.rs.ext.RuntimeDelegate
> code-source
> /usr/bigtop/3.2.0/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar
> name javax.ws.rs.ext.RuntimeDelegate
> isInterface false
> isAnnotation false
> isEnum false
> isAnonymousClass false
> isArray false
> isLocalClass false
> isMemberClass false
> isPrimitive false
> isSynthetic false
> simple-name RuntimeDelegate
> modifier abstract,public
> annotation
> interfaces
> super-class +-java.lang.Object
> class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
> +-sun.misc.Launcher$ExtClassLoader@436a4e4b
> classLoaderHash 4fca772dAffect(row-cnt:3) cost in 112 ms. {code}
> By removing the jersey-core-1.19.3.jar file from
> /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/, we can see
> that the abstract class javax.ws.rs.ext.RuntimeDelegate is loaded by the
> AppClassLoader from the javax.ws.rs-api-2.1.1.jar file located in
> /usr/bigtop/3.2.0/usr/lib/hbase/lib/.
> At the same time, the implementation of javax.ws.rs.ext.RuntimeDelegate,
> which is com.sun.jersey.server.impl.provider.RuntimeDelegateImpl, is loaded
> from the jersey-server-1.19.jar file located in
> /usr/bigtop/3.2.0/usr/lib/hadoop/lib/. Note that the hadoop lib directory is
> appended to HBase's classpath when HBase is started, which is why it can be
> loaded.
> Additionally, the abstract class
> com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate, which inherits from
> RuntimeDelegate, is loaded from the jersey-core-1.19.jar file located in
> /usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/.
> However, even after removing the jersey-core-1.19.3.jar file, we can still
> see that HBase loads the same version of jersey-core-1.19.jar from the hadoop
> lib directory. Therefore, the root cause of the error is not a class not
> found due to a version conflict, but rather due to the order in which the
> class loader loads the classes.
>
> To verify this conclusion, the simplest way is to:
> # Before deleting the jersey-core-1.19.3.jar file from
> /ranger-hbase-plugin-impl/, print out the loading information of all the
> classes involved in the stack trace of the heap dump exception in the log.
> # After deleting the jersey-core-1.19.3.jar file, print out the loading
> information of all the classes involved in the stack trace of the heap dump
> exception in the log.
> # Use regular expressions to batch delete the timestamps from the log, and
> then compare the two log files to see any differences in the class loading
> information.
> In the Ranger 2.4 source code, Ranger implements its own class loader,
> RangerPluginClassLoader, which inherits from URLClassLoader. During
> initialization, it is passed the path to the
> ranger-\{componentname}{-}plugin-impl directory, and all the classes in this
> directory will be loaded by RangerPluginClassLoader. The parent is set to
> null, which in my understanding, is to block delegation loading. This is
> because all the classes in the ranger{-}{componentname}-plugin-impl directory
> are dependencies of the Ranger plugin. For example, if the version of a jar
> that it depends on is lower, if the delegation loading mechanism is used, it
> is easy to load the higher version of the class under the component/lib
> directory, leading to class conflict issues. Therefore, parent is set to null.
> Internally, RangerPluginClassLoader first tries to load the jar files under
> the ranger-\{componentname}-plugin-impl directory using the parent
> URLClassLoader. If it can't be loaded, it uses the componentClassLoader,
> which is the current thread context's class loader, to load it.
> !image-2023-04-27-16-13-44-504.png!
> Next, you can take a look at the source code of the
> "RangerAuthorizationCoprocessor" class in Ranger. This class initializes an
> instance of "rangerPluginClassLoader" and activates it. Within this class,
> there are pre and post hooks for various HBase operations, primarily for
> performing range permission-related operations. This triggers the loading of
> Ranger-related classes by the rangerPluginClassLoader.
> !image-2023-04-27-16-14-10-803.png!
> Why does the hook activate the Ranger class loader at the beginning and
> de-activate it after the hook method ends? Doesn't this approach impact
> performance?
> !image-2023-04-27-16-14-37-202.png!
> The reason for de-activating the Ranger class loader after the hook method
> ends is because the rangerPluginClassLoader code sets itself as the current
> thread's context class loader. If it's not deactivated, it can cause
> conflicts with the lower version packages loaded in the
> .../ranger-hbase-plugin-impl/ directory, which can cause issues during
> subsequent operations. Therefore, the conclusion is that this involves the
> rangerPluginClassLoader and the AppClassLoader of the thread's context class
> loader.
> We also need to understand how to load an instance of "RuntimeDelegate" and
> how it searches for the implementation of "javax.ws.rs.ext.RuntimeDelegate".
> We should also investigate why it cannot find the implementation class and
> instead looks for "org.glassfish.jersey.internal.RuntimeDelegateImpl",
> causing an error.
> First, the "RuntimeDelegate" class is loaded via the
> "javax.ws.rs.ext.RuntimeDelegate.findDelegate" method, which calls
> "FactoryFinder.find" to load the class. Once loaded, the instance must be of
> type "RuntimeDelegate".
> {code:java}
> private static RuntimeDelegate findDelegate() {
> try {
> Object delegate = FactoryFinder.find("javax.ws.rs.ext.RuntimeDelegate",
> "org.glassfish.jersey.internal.RuntimeDelegateImpl", RuntimeDelegate.class);
> if (!(delegate instanceof RuntimeDelegate)) {
> Class<RuntimeDelegate> pClass = RuntimeDelegate.class;
> String classnameAsResource = pClass.getName().replace('.', '/') +
> ".class";
> ClassLoader loader = pClass.getClassLoader();
> if (loader == null)
> loader = ClassLoader.getSystemClassLoader();
> URL targetTypeURL = loader.getResource(classnameAsResource);
> throw new LinkageError("ClassCastException: attempting to cast" +
> delegate
> .getClass().getClassLoader().getResource(classnameAsResource) + "
> to " + targetTypeURL);
> }
> return (RuntimeDelegate)delegate;
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> } {code}
> The process of loading the class via "FactoryFinder.find" is contained in the
> "javax.ws.rs.ext.FactoryFinder.find" method, as shown in the following code:
> {code:java}
> /**
> * Finds the implementation {@code Class} for the given factory name,
> * or if that fails, finds the {@code Class} for the given fallback
> * class name and create its instance. The arguments supplied MUST be
> * used in order. If using the first argument is successful, the second
> * one will not be used.
> * <p>
> * This method is package private so that this code can be shared.
> *
> * @param factoryId the name of the factory to find, which is
> * a system property.
> * @param fallbackClassName the implementation class name, which is
> * to be used only if nothing else.
> * is found; {@code null} to indicate that
> * there is no fallback class name.
> * @param service service to be found.
> * @param <T> type of the service to be found.
> * @return the instance of the specified service; may not be {@code null}.
> * @throws ClassNotFoundException if the given class could not be found
> * or could not be instantiated.
> */
> static <T> Object find(final String factoryId, final String
> fallbackClassName, Class<T> service) throws ClassNotFoundException {
> ClassLoader classLoader = getContextClassLoader(); try {
> Iterator<T> iterator = ServiceLoader.load(service,
> FactoryFinder.getContextClassLoader()).iterator();
> if(iterator.hasNext()) {
> return iterator.next();
> }
> } catch (Exception | ServiceConfigurationError ex) {
> LOGGER.log(Level.FINER, "Failed to load service " + factoryId +
> ".", ex);
> } try {
> Iterator<T> iterator = ServiceLoader.load(service,
> FactoryFinder.class.getClassLoader()).iterator();
> if(iterator.hasNext()) {
> return iterator.next();
> }
> } catch (Exception | ServiceConfigurationError ex) {
> LOGGER.log(Level.FINER, "Failed to load service " + factoryId +
> ".", ex);
> } // try to read from $java.home/lib/jaxrs.properties
> FileInputStream inputStream = null;
> String configFile = null;
> try {
> String javah = System.getProperty("java.home");
> configFile = javah + File.separator + "lib" + File.separator +
> "jaxrs.properties";
> File f = new File(configFile);
> if (f.exists()) {
> Properties props = new Properties();
> inputStream = new FileInputStream(f);
> props.load(inputStream);
> String factoryClassName = props.getProperty(factoryId);
> return newInstance(factoryClassName, classLoader);
> }
> } catch (Exception ex) {
> LOGGER.log(Level.FINER, "Failed to load service " + factoryId
> + " from $java.home/lib/jaxrs.properties", ex);
> } finally {
> if (inputStream != null) {
> try {
> inputStream.close();
> } catch (IOException ex) {
> LOGGER.log(Level.FINER, String.format("Error closing %s
> file.", configFile), ex);
> }
> }
> } // Use the system property
> try {
> String systemProp = System.getProperty(factoryId);
> if (systemProp != null) {
> return newInstance(systemProp, classLoader);
> }
> } catch (SecurityException se) {
> LOGGER.log(Level.FINER, "Failed to load service " + factoryId
> + " from a system property", se);
> } if (fallbackClassName == null) {
> throw new ClassNotFoundException(
> "Provider for " + factoryId + " cannot be found", null);
> } return newInstance(fallbackClassName, classLoader);
> }
> } {code}
> To summarize the "FactoryFinder.find" method, it uses the thread context
> class loader to load "javax.ws.rs.ext.RuntimeDelegate". If that fails, it
> attempts to load it from system properties or alternative class names. If it
> still cannot be found, it tries to load
> "org.glassfish.jersey.internal.RuntimeDelegateImpl". If it cannot be loaded,
> it throws an exception.
> When the
> /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar
> file was not deleted, the class loading order was as follows:
> javax.ws.rs.ext.RuntimeDelegate(appClassLoader) ->
> javax.ws.rs.ext.RuntimeDelegate.HeaderDelegate(RangerPluginClassLoader) ->
> javax.ws.rs.ext.RuntimeDelegate.HeaderDelegateProvider(RangerPluginClassLoader
> jersey-core-1.19.3.jar ranger-hbase-plugin-impl/jersey-core-1.19.3.jar).
> The RangerPluginClassLoader searched for the
> com.sun.jersey.spi.HeaderDelegateProvider implementation class in the
> META-INF/services/com.sun.jersey.spi.HeaderDelegateProvider file of the
> jersey-core-1.19.3.jar file, found the
> com.sun.jersey.core.impl.provider.header.LocaleProvider class in
> ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, and returned it.
> However, javax.ws.rs.ext.RuntimeDelegate.findDelegate would fail because
> delegate instanceof RuntimeDelegate returned false.
> As a fallback, it searched for
> org.glassfish.jersey.internal.RuntimeDelegateImpl, which was not found in the
> hbase classpath and caused an exception to be thrown.
> After removing ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, HBase can be
> started normally. The class loading order is as follows:
> javax.ws.rs.ext.RuntimeDelegate is loaded from
> /usr/lib/hadoop/lib/javax.ws.rs-api-2.1.1.jar. Then, the implementation class
> of javax.ws.rs.ext.RuntimeDelegate is searched for, which is found in
> /usr/lib/hadoop/lib/jersey-server-1.19.jar->META-INF/services/javax.ws.rs.ext.RuntimeDelegate->com.sun.jersey.server.impl.provider.RuntimeDelegateImpl,
> and successfully loaded.
> The above class loading order was determined by modifying the code in the
> RangerPluginClassLoader to print out the process of loading all classes.
> !image-2023-04-27-16-15-44-539.png!
> The following is a log that includes the class loading order. By analyzing
> this log in conjunction with the Ranger class loader code, the above results
> can be determined.
> [^hbase-hbase-master.err.log]
> [^hbase-hbase-master.log]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
