[ https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323051#comment-16323051 ]
stack commented on HBASE-19769: ------------------------------- We don't want to use hadoops metrics long-term if thats of any use to. Its only been a pain. Modules for hadoop1 vs hadoop2. We have hacks in place to kill it and restart it because no other way of onlining new metrics -- e.g. region added/removed. We spend most of our CPU counting so we should probably own the metrics system rather than defer to another. #1 sounds easy-to-do? Nice to have but an aspect of general client metrics and user is not completely lost in that they can look at zk ensemble metrics. #1 has never shipped in any hbase? > IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce > jobs > ------------------------------------------------------------------------------- > > Key: HBASE-19769 > URL: https://issues.apache.org/jira/browse/HBASE-19769 > Project: HBase > Issue Type: Bug > Components: mapreduce, metrics > Affects Versions: 2.0.0-beta-1 > Reporter: Josh Elser > Assignee: Josh Elser > Priority: Critical > Fix For: 2.0.0-beta-2 > > > issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893 > Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has > been using a custom classloader to separate Hadoop dependencies from the > user's JAR being run. A separate classloader is created for the user-provided > jar, and then this classloader is set as the contextClassLoader before the > Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for > us to date because we don't try to access any Hadoop internal classes > client-side. > However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The > problem is that Hadoop metrics2 implementations which we reference from the > same package are loaded by a different classloader than our HBase code is > loaded from. This makes the expected package-private access of these Metrics2 > classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError. > {noformat} > java.lang.RuntimeException: Could not create interface > org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop > compatibility jar on the classpath? > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75) > at > org.apache.hadoop.hbase.zookeeper.ZKMetrics.<init>(ZKMetrics.java:36) > at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:115) > at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139) > at > org.apache.hadoop.hbase.zookeeper.ZKWatcher.<init>(ZKWatcher.java:128) > at > org.apache.hadoop.hbase.zookeeper.ZKWatcher.<init>(ZKWatcher.java:102) > at > org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293) > at > org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259) > at > org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535) > at > org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712) > at > org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300) > at > org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267) > at > org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) > at > org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:239) > at org.apache.hadoop.util.RunJar.main(RunJar.java:153) > Caused by: java.util.ServiceConfigurationError: > org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider > org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be > instantiated > at java.util.ServiceLoader.fail(ServiceLoader.java:232) > at java.util.ServiceLoader.access$100(ServiceLoader.java:185) > at > java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) > at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) > at java.util.ServiceLoader$1.next(ServiceLoader.java:480) > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59) > ... 21 more > Caused by: java.lang.IllegalAccessError: tried to access class > org.apache.hadoop.metrics2.lib.MetricsInfoImpl from class > org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry > at > org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry.newGauge(DynamicMetricsRegistry.java:139) > at > org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.<init>(MetricsZooKeeperSourceImpl.java:59) > at > org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.<init>(MetricsZooKeeperSourceImpl.java:51) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at java.lang.Class.newInstance(Class.java:442) > at > java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) > ... 24 more > {noformat} > There are two ways to fix this problem that I see now: > # Remove client-side ZK metrics and disallow any more client-side metrics > that directly use metrics2 API > # Use public API for metrics2 only in HBase, or copy implementations from > Hadoop to avoid the need to access the package-private classes from this > goofy state. -- This message was sent by Atlassian JIRA (v6.4.14#64029)