[
https://issues.apache.org/jira/browse/RANGER-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caijialiang updated RANGER-4201:
--------------------------------
Attachment: hbase-hbase-master.log
> Hbase master can't start due to ranger-hbase-plugin jersey jar class loading
> order
> -----------------------------------------------------------------------------------
>
> Key: RANGER-4201
> URL: https://issues.apache.org/jira/browse/RANGER-4201
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
> Affects Versions: 2.3.0, 2.4.0
> Reporter: caijialiang
> Assignee: caijialiang
> Priority: Major
> Attachments: hbase-hbase-master.err.log, hbase-hbase-master.log,
> image-2023-04-27-16-11-05-561.png, image-2023-04-27-16-13-44-504.png,
> image-2023-04-27-16-14-10-803.png, image-2023-04-27-16-14-37-202.png,
> image-2023-04-27-16-15-44-539.png
>
>
>
> Issue description: HBase fails to start after enabling Ranger 2.4 plugin.
> Problem analysis is required.
> environment：
> linux centos7.4
> hbase 2.4.13
> ranger 2.4
> jdk1.8
> hadoop 3.3.4
> zookeeper 3.5.9
> Symptom: After integrating Ranger with HBase, HBase Master cannot start, and
> Region Server also crashes after running for a period of time. The same error
> message occurs during both processes.
>
> {code:java}
> [master/gs-server-13481:16000:becomeActiveMaster]
> coprocessor.CoprocessorHost: The coprocessor
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor threw
> java.lang.ExceptionInInitializerError
> java.lang.ExceptionInInitializerError
> at
> com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182)
> at
> com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175)
> at
> com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)
> at com.sun.jersey.api.client.Client.init(Client.java:343)
> at com.sun.jersey.api.client.Client.access$000(Client.java:119)
> at com.sun.jersey.api.client.Client$1.f(Client.java:192)
> at com.sun.jersey.api.client.Client$1.f(Client.java:188)
> at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
> at com.sun.jersey.api.client.Client.<init>(Client.java:188)
> at com.sun.jersey.api.client.Client.<init>(Client.java:171)
> at com.sun.jersey.api.client.Client.create(Client.java:683)
> at
> org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:228)
> at
> org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:193)
> at
> org.apache.ranger.plugin.util.RangerRESTClient.get(RangerRESTClient.java:473)
> at
> org.apache.ranger.admin.client.RangerAdminRESTClient.getRangerRolesDownloadResponse(RangerAdminRESTClient.java:1340)
> at
> org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1202)
> at
> org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:167)
> at
> org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:183)
> at
> org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:123)
> at
> org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:495)
> at
> org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:144)
> at
> org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:245)
> at
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:1120)
> at
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start(RangerAuthorizationCoprocessor.java:160)
> at
> org.apache.hadoop.hbase.coprocessor.BaseEnvironment.startup(BaseEnvironment.java:69)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.checkAndLoadInstance(CoprocessorHost.java:285)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:171)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.<init>(MasterCoprocessorHost.java:155)
> at
> org.apache.hadoop.hbase.master.HMaster.initializeCoprocessorHost(HMaster.java:3870)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:902)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2175)
> at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:520)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.glassfish.jersey.internal.RuntimeDelegateImpl
> at
> javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:130)
> at
> javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:97)
> at javax.ws.rs.core.MediaType.valueOf(MediaType.java:172)
> at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:65)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException:
> org.glassfish.jersey.internal.RuntimeDelegateImpl
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:87)
> at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:185)
> at
> javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:112)
> ... 36 more {code}
> Steps to reproduce:
>
> * 1.After installing HBase 2.x and Ranger 2.4, configure HBase by modifying
> the hbase-site.xml file:
> * Set hbase.coprocessor.master.classes to
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
> * Set hbase.coprocessor.region.classes to
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
> * Set hbase.coprocessor.regionserver.classes to
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor
> * Restart HBase and check the HBase Master logs to observe the exception and
> see the Master crash soon after
> solution：delete
> /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar
>
>
> related issues
> https://issues.apache.org/jira/browse/HBASE-22029
> https://issues.apache.org/jira/browse/HBASE-22052
> !image-2023-04-27-16-11-05-561.png!
> Problem Analysis:
> Conclusion: Exception caused by class loading order
>
> Analysis process: The call chain is roughly as follows: master start {-}>
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.start
> ... ->com.sun.jersey.api.client.Client.create
> ...{-}>com.sun.jersey.core.spi.factory.MessageBodyFactory.init
> ->com.sun.jersey.core.header.MediaTypes ->javax.ws.rs.core.MediaType.valueOf
> -> javax.ws.rs.ext.RuntimeDelegate.getInstance ->
> javax.ws.rs.ext.RuntimeDelegate.findDelegate
> I have fully investigated this call chain and found that the root cause of
> the problem is that javax.ws.rs.ext.RuntimeDelegate in the
> javax.ws.rs-api-2.1.1.jar package in HBase did not find
> org.glassfish.jersey.internal.RuntimeDelegateImpl when calling findDelegate,
> which caused the error. Specifically, javax.ws.rs.ext.RuntimeDelegate first
> looks for the default JAXRS_RUNTIME_DELEGATE, which is
> "javax.ws.rs.ext.RuntimeDelegate". If it cannot be found, it will fallback
> and load JAXRS_DEFAULT_RUNTIME_DELEGATE, which is
> org.glassfish.jersey.internal.RuntimeDelegateImpl.
> The method "findDelegate" in "javax.ws.rs.ext.RuntimeDelegate" is as follows:
> {code:java}
> public abstract class RuntimeDelegate {
> public static final String JAXRS_RUNTIME_DELEGATE_PROPERTY =
> "javax.ws.rs.ext.RuntimeDelegate";
> private static final String JAXRS_DEFAULT_RUNTIME_DELEGATE =
> "org.glassfish.jersey.internal.RuntimeDelegateImpl";
> private static final Object RD_LOCK = new Object();
> private static ReflectPermission suppressAccessChecksPermission = new
> ReflectPermission("suppressAccessChecks");
> private static volatile RuntimeDelegate cachedDelegate; protected
> RuntimeDelegate() {
> }
> public static RuntimeDelegate getInstance() {
> // Double-check idiom for lazy initialization of fields.
> // Local variable is used to limit the number of more expensive
> accesses to a volatile field.
> RuntimeDelegate result = cachedDelegate;
> if (result == null) { // First check (no locking)
> synchronized (RD_LOCK) {
> result = cachedDelegate;
> if (result == null) { // Second check (with locking)
> cachedDelegate = result = findDelegate();
> }
> }
> }
> return result;
> } private static RuntimeDelegate findDelegate() {
> try {
> Object delegate = FactoryFinder.find(
> JAXRS_RUNTIME_DELEGATE_PROPERTY,
> JAXRS_DEFAULT_RUNTIME_DELEGATE,
> RuntimeDelegate.class);
> if (!(delegate instanceof RuntimeDelegate)) {
> Class pClass = RuntimeDelegate.class;
> String classnameAsResource = pClass.getName().replace('.',
> '/') + ".class";
> ClassLoader loader = pClass.getClassLoader();
> if (loader == null) {
> loader = ClassLoader.getSystemClassLoader();
> }
> URL targetTypeURL = loader.getResource(classnameAsResource);
> throw new LinkageError("ClassCastException: attempting to
> cast"
> +
> delegate.getClass().getClassLoader().getResource(classnameAsResource)
> + " to " + targetTypeURL);
> }
> return (RuntimeDelegate) delegate;
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> }
> {code}
> When the "jersey-core-1.19.3.jar" file is removed from
> "/usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/", HBase can
> start normally.
>
> It is not clear which class, either "javax.ws.rs.ext.RuntimeDelegate" or
> "org.glassfish.jersey.internal.RuntimeDelegateImpl", is loaded by HBase at
> this point.
>
> To determine which class is loaded and from which JAR file it is loaded, we
> can first remove "/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar" and start
> HBase normally. Then, we can use the Arthas tool
> ([https://github.com/alibaba/arthas]) to sniff and determine which class is
> loaded and from which JAR file it is loaded by the HBase JVM.
> After starting HBase normally, run the Arthas startup script "sudo -u hbase
> -EH ./as.sh " attach to the HBase process. Then, execute "sc -d
> javax.ws.rs.ext.RuntimeDelegat" to check whether
> "javax.ws.rs.ext.RuntimeDelegate" is loaded.
>
> result：
> {code:java}
> [arthas@1719]$ sc -d javax.ws.rs.ext.RuntimeDelegate
> class-info com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
> code-source
> /usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/jersey-core-1.19.jar
> name com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
> isInterface false
> isAnnotation false
> isEnum false
> isAnonymousClass false
> isArray false
> isLocalClass false
> isMemberClass false
> isPrimitive false
> isSynthetic false
> simple-name AbstractRuntimeDelegate
> modifier abstract,public
> annotation
> interfaces
> super-class +-javax.ws.rs.ext.RuntimeDelegate
> +-java.lang.Object
> class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
> +-sun.misc.Launcher$ExtClassLoader@436a4e4b
> classLoaderHash 4fca772d class-info
> com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
> code-source /usr/bigtop/3.2.0/usr/lib/hadoop/lib/jersey-server-1.19.jar
> name com.sun.jersey.server.impl.provider.RuntimeDelegateImpl
> isInterface false
> isAnnotation false
> isEnum false
> isAnonymousClass false
> isArray false
> isLocalClass false
> isMemberClass false
> isPrimitive false
> isSynthetic false
> simple-name RuntimeDelegateImpl
> modifier public
> annotation
> interfaces
> super-class +-com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate
> +-javax.ws.rs.ext.RuntimeDelegate
> +-java.lang.Object
> class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
> +-sun.misc.Launcher$ExtClassLoader@436a4e4b
> classLoaderHash 4fca772d class-info javax.ws.rs.ext.RuntimeDelegate
> code-source
> /usr/bigtop/3.2.0/usr/lib/hbase/lib/javax.ws.rs-api-2.1.1.jar
> name javax.ws.rs.ext.RuntimeDelegate
> isInterface false
> isAnnotation false
> isEnum false
> isAnonymousClass false
> isArray false
> isLocalClass false
> isMemberClass false
> isPrimitive false
> isSynthetic false
> simple-name RuntimeDelegate
> modifier abstract,public
> annotation
> interfaces
> super-class +-java.lang.Object
> class-loader +-sun.misc.Launcher$AppClassLoader@4fca772d
> +-sun.misc.Launcher$ExtClassLoader@436a4e4b
> classLoaderHash 4fca772dAffect(row-cnt:3) cost in 112 ms. {code}
> By removing the jersey-core-1.19.3.jar file from
> /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/, we can see
> that the abstract class javax.ws.rs.ext.RuntimeDelegate is loaded by the
> AppClassLoader from the javax.ws.rs-api-2.1.1.jar file located in
> /usr/bigtop/3.2.0/usr/lib/hbase/lib/.
> At the same time, the implementation of javax.ws.rs.ext.RuntimeDelegate,
> which is com.sun.jersey.server.impl.provider.RuntimeDelegateImpl, is loaded
> from the jersey-server-1.19.jar file located in
> /usr/bigtop/3.2.0/usr/lib/hadoop/lib/. Note that the hadoop lib directory is
> appended to HBase's classpath when HBase is started, which is why it can be
> loaded.
> Additionally, the abstract class
> com.sun.jersey.core.spi.factory.AbstractRuntimeDelegate, which inherits from
> RuntimeDelegate, is loaded from the jersey-core-1.19.jar file located in
> /usr/bigtop/3.2.0/usr/lib/hadoop-hdfs/lib/.
> However, even after removing the jersey-core-1.19.3.jar file, we can still
> see that HBase loads the same version of jersey-core-1.19.jar from the hadoop
> lib directory. Therefore, the root cause of the error is not a class not
> found due to a version conflict, but rather due to the order in which the
> class loader loads the classes.
>
> To verify this conclusion, the simplest way is to:
> # Before deleting the jersey-core-1.19.3.jar file from
> /ranger-hbase-plugin-impl/, print out the loading information of all the
> classes involved in the stack trace of the heap dump exception in the log.
> # After deleting the jersey-core-1.19.3.jar file, print out the loading
> information of all the classes involved in the stack trace of the heap dump
> exception in the log.
> # Use regular expressions to batch delete the timestamps from the log, and
> then compare the two log files to see any differences in the class loading
> information.
> In the Ranger 2.4 source code, Ranger implements its own class loader,
> RangerPluginClassLoader, which inherits from URLClassLoader. During
> initialization, it is passed the path to the
> ranger-\{componentname}{-}plugin-impl directory, and all the classes in this
> directory will be loaded by RangerPluginClassLoader. The parent is set to
> null, which in my understanding, is to block delegation loading. This is
> because all the classes in the ranger{-}{componentname}-plugin-impl directory
> are dependencies of the Ranger plugin. For example, if the version of a jar
> that it depends on is lower, if the delegation loading mechanism is used, it
> is easy to load the higher version of the class under the component/lib
> directory, leading to class conflict issues. Therefore, parent is set to null.
> Internally, RangerPluginClassLoader first tries to load the jar files under
> the ranger-\{componentname}-plugin-impl directory using the parent
> URLClassLoader. If it can't be loaded, it uses the componentClassLoader,
> which is the current thread context's class loader, to load it.
> !image-2023-04-27-16-13-44-504.png!
> Next, you can take a look at the source code of the
> "RangerAuthorizationCoprocessor" class in Ranger. This class initializes an
> instance of "rangerPluginClassLoader" and activates it. Within this class,
> there are pre and post hooks for various HBase operations, primarily for
> performing range permission-related operations. This triggers the loading of
> Ranger-related classes by the rangerPluginClassLoader.
> !image-2023-04-27-16-14-10-803.png!
> Why does the hook activate the Ranger class loader at the beginning and
> de-activate it after the hook method ends? Doesn't this approach impact
> performance?
> !image-2023-04-27-16-14-37-202.png!
> The reason for de-activating the Ranger class loader after the hook method
> ends is because the rangerPluginClassLoader code sets itself as the current
> thread's context class loader. If it's not deactivated, it can cause
> conflicts with the lower version packages loaded in the
> .../ranger-hbase-plugin-impl/ directory, which can cause issues during
> subsequent operations. Therefore, the conclusion is that this involves the
> rangerPluginClassLoader and the AppClassLoader of the thread's context class
> loader.
> We also need to understand how to load an instance of "RuntimeDelegate" and
> how it searches for the implementation of "javax.ws.rs.ext.RuntimeDelegate".
> We should also investigate why it cannot find the implementation class and
> instead looks for "org.glassfish.jersey.internal.RuntimeDelegateImpl",
> causing an error.
> First, the "RuntimeDelegate" class is loaded via the
> "javax.ws.rs.ext.RuntimeDelegate.findDelegate" method, which calls
> "FactoryFinder.find" to load the class. Once loaded, the instance must be of
> type "RuntimeDelegate".
> {code:java}
> private static RuntimeDelegate findDelegate() {
> try {
> Object delegate = FactoryFinder.find("javax.ws.rs.ext.RuntimeDelegate",
> "org.glassfish.jersey.internal.RuntimeDelegateImpl", RuntimeDelegate.class);
> if (!(delegate instanceof RuntimeDelegate)) {
> Class<RuntimeDelegate> pClass = RuntimeDelegate.class;
> String classnameAsResource = pClass.getName().replace('.', '/') +
> ".class";
> ClassLoader loader = pClass.getClassLoader();
> if (loader == null)
> loader = ClassLoader.getSystemClassLoader();
> URL targetTypeURL = loader.getResource(classnameAsResource);
> throw new LinkageError("ClassCastException: attempting to cast" +
> delegate
> .getClass().getClassLoader().getResource(classnameAsResource) + "
> to " + targetTypeURL);
> }
> return (RuntimeDelegate)delegate;
> } catch (Exception ex) {
> throw new RuntimeException(ex);
> }
> } {code}
> The process of loading the class via "FactoryFinder.find" is contained in the
> "javax.ws.rs.ext.FactoryFinder.find" method, as shown in the following code:
> {code:java}
> /**
> * Finds the implementation {@code Class} for the given factory name,
> * or if that fails, finds the {@code Class} for the given fallback
> * class name and create its instance. The arguments supplied MUST be
> * used in order. If using the first argument is successful, the second
> * one will not be used.
> * <p>
> * This method is package private so that this code can be shared.
> *
> * @param factoryId the name of the factory to find, which is
> * a system property.
> * @param fallbackClassName the implementation class name, which is
> * to be used only if nothing else.
> * is found; {@code null} to indicate that
> * there is no fallback class name.
> * @param service service to be found.
> * @param <T> type of the service to be found.
> * @return the instance of the specified service; may not be {@code null}.
> * @throws ClassNotFoundException if the given class could not be found
> * or could not be instantiated.
> */
> static <T> Object find(final String factoryId, final String
> fallbackClassName, Class<T> service) throws ClassNotFoundException {
> ClassLoader classLoader = getContextClassLoader(); try {
> Iterator<T> iterator = ServiceLoader.load(service,
> FactoryFinder.getContextClassLoader()).iterator();
> if(iterator.hasNext()) {
> return iterator.next();
> }
> } catch (Exception | ServiceConfigurationError ex) {
> LOGGER.log(Level.FINER, "Failed to load service " + factoryId +
> ".", ex);
> } try {
> Iterator<T> iterator = ServiceLoader.load(service,
> FactoryFinder.class.getClassLoader()).iterator();
> if(iterator.hasNext()) {
> return iterator.next();
> }
> } catch (Exception | ServiceConfigurationError ex) {
> LOGGER.log(Level.FINER, "Failed to load service " + factoryId +
> ".", ex);
> } // try to read from $java.home/lib/jaxrs.properties
> FileInputStream inputStream = null;
> String configFile = null;
> try {
> String javah = System.getProperty("java.home");
> configFile = javah + File.separator + "lib" + File.separator +
> "jaxrs.properties";
> File f = new File(configFile);
> if (f.exists()) {
> Properties props = new Properties();
> inputStream = new FileInputStream(f);
> props.load(inputStream);
> String factoryClassName = props.getProperty(factoryId);
> return newInstance(factoryClassName, classLoader);
> }
> } catch (Exception ex) {
> LOGGER.log(Level.FINER, "Failed to load service " + factoryId
> + " from $java.home/lib/jaxrs.properties", ex);
> } finally {
> if (inputStream != null) {
> try {
> inputStream.close();
> } catch (IOException ex) {
> LOGGER.log(Level.FINER, String.format("Error closing %s
> file.", configFile), ex);
> }
> }
> } // Use the system property
> try {
> String systemProp = System.getProperty(factoryId);
> if (systemProp != null) {
> return newInstance(systemProp, classLoader);
> }
> } catch (SecurityException se) {
> LOGGER.log(Level.FINER, "Failed to load service " + factoryId
> + " from a system property", se);
> } if (fallbackClassName == null) {
> throw new ClassNotFoundException(
> "Provider for " + factoryId + " cannot be found", null);
> } return newInstance(fallbackClassName, classLoader);
> }
> } {code}
> To summarize the "FactoryFinder.find" method, it uses the thread context
> class loader to load "javax.ws.rs.ext.RuntimeDelegate". If that fails, it
> attempts to load it from system properties or alternative class names. If it
> still cannot be found, it tries to load
> "org.glassfish.jersey.internal.RuntimeDelegateImpl". If it cannot be loaded,
> it throws an exception.
> When the
> /usr/bigtop/3.2.0/usr/lib/hbase/lib/ranger-hbase-plugin-impl/jersey-core-1.19.3.jar
> file was not deleted, the class loading order was as follows:
> javax.ws.rs.ext.RuntimeDelegate(appClassLoader) ->
> javax.ws.rs.ext.RuntimeDelegate.HeaderDelegate(RangerPluginClassLoader) ->
> javax.ws.rs.ext.RuntimeDelegate.HeaderDelegateProvider(RangerPluginClassLoader
> jersey-core-1.19.3.jar ranger-hbase-plugin-impl/jersey-core-1.19.3.jar).
> The RangerPluginClassLoader searched for the
> com.sun.jersey.spi.HeaderDelegateProvider implementation class in the
> META-INF/services/com.sun.jersey.spi.HeaderDelegateProvider file of the
> jersey-core-1.19.3.jar file, found the
> com.sun.jersey.core.impl.provider.header.LocaleProvider class in
> ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, and returned it.
> However, javax.ws.rs.ext.RuntimeDelegate.findDelegate would fail because
> delegate instanceof RuntimeDelegate returned false.
> As a fallback, it searched for
> org.glassfish.jersey.internal.RuntimeDelegateImpl, which was not found in the
> hbase classpath and caused an exception to be thrown.
> After removing ranger-hbase-plugin-impl/jersey-core-1.19.3.jar, HBase can be
> started normally. The class loading order is as follows:
> javax.ws.rs.ext.RuntimeDelegate is loaded from
> /usr/lib/hadoop/lib/javax.ws.rs-api-2.1.1.jar. Then, the implementation class
> of javax.ws.rs.ext.RuntimeDelegate is searched for, which is found in
> /usr/lib/hadoop/lib/jersey-server-1.19.jar->META-INF/services/javax.ws.rs.ext.RuntimeDelegate->com.sun.jersey.server.impl.provider.RuntimeDelegateImpl,
> and successfully loaded.
> The above class loading order was determined by modifying the code in the
> RangerPluginClassLoader to print out the process of loading all classes.
> !image-2023-04-27-16-15-44-539.png!
> The following is a log that includes the class loading order. By analyzing
> this log in conjunction with the Ranger class loader code, the above results
> can be determined.
> [^hbase-hbase-master.err.log]
> [^hbase-hbase-master.log]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
