[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383348#comment-16383348
]
Hudson commented on HBASE-19769:
Results for branch branch-1, done in 47 min and counting
[build #236 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/236/]:
details (if available):
(x) *{color:red}-1 overall{color}*
Committer, please check your recent inclusion of a patch for this issue.
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 1.5.0, 2.0.0-beta-2, 1.4.3
>
> Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.hbase.Compat
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382487#comment-16382487
]
Andrew Purtell commented on HBASE-19769:
Ok. Let me revert the commit of HBASE-17448 on branch-1 and branch-1.4,
referencing this JIRA, and state that the plan is to eventually reimplement
with hbase-metrics-api or after HBASE-18409 gets in
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
> ... 21 more
> Caused by: java.lang.IllegalAccessError: tried to access class
> org.apache.hadoop.me
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382360#comment-16382360
]
stack commented on HBASE-19769:
---
bq. For branch-2, use the hbase-metrics API and avoid metrics2 completely (I
think helping HBASE-18409 get in would be helpful, maybe necessary)
Arg... Should have kept an eye on HBASE-18409 Thats a shame. Its too
late now... but hadoop metrics are only ever going to be pain producing.
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
> ... 21 more
> Caused by:
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382283#comment-16382283
]
Josh Elser commented on HBASE-19769:
{quote}Would have been useful to get the reports about issues regarding
HBASE-17448 expressed over there during review, or at least expressed over
there when this got pulled from 2.0. I see the following:
{quote}
Sorry, am watching it now.
{quote}I feel this should come out in 1.5 since it is already out in 2.0, at
least.
{quote}
Yeah, I'd agree with you.
{quote}Do you know if this an issue in 1.4x [~elserj] also sir?
{quote}
I haven't explicitly tested it, but I would assume the issue also exists there.
Looking at DynamicMetricsRegistry, I see we're referencing MetricsInfoImpl
which was the problematic class being accessed across these ClassLoader
boundaries.
{quote}Although this shipped in 1.4.0, 1.4.1, and 1.4.2, I have no problem
pulling it from 1.5 if problematic, or even from 1.4.3 if more than problematic
approaching broken. Please advise.
{quote}
Yeah, pulling from 1.4 would probably be best also (I'm not sure why I didn't
ask the question then, apologies). I think there are two potential fixes:
# For branch-1, lift anything package-private from metrics2 we have to rely on
into hbase itself
# For branch-2, use the hbase-metrics API and avoid metrics2 completely (I
think helping HBASE-18409 get in would be helpful, maybe necessary)
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessor
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381524#comment-16381524
]
stack commented on HBASE-19769:
---
bq. Would have been useful to get the reports about issues regarding
HBASE-17448 expressed over there during review, or at least expressed over
there when this got pulled from 2.0.
Yes. Should have updated the original issue [~apurtell].
Do you know if this an issue in 1.4x [~elserj] also sir?
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
> ... 21 more
> Caused by:
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381180#comment-16381180
]
Andrew Purtell commented on HBASE-19769:
I feel this should come out in 1.5 since it is already out in 2.0, at least.
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
> ... 21 more
> Caused by: java.lang.IllegalAccessError: tried to access class
> org.apache.hadoop.metrics2.lib.MetricsInfoImpl from class
> org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry
> at
> org.apache.hadoop
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381173#comment-16381173
]
Andrew Purtell commented on HBASE-19769:
Would have been useful to get the reports about issues regarding HBASE-17448
expressed over there during review, or at least expressed over there when this
got pulled from 2.0. I see the following:
- Issues with service loading in YARN
- Too verbose (? - they are metrics)
- Dont' want stuff based on hadoop-metrics
All valid.
Although this shipped in 1.4.0, 1.4.1, and 1.4.2, I have no problem pulling it
from 1.5 if problematic, or even from 1.4.3 if more than problematic
approaching broken. Please advise.
[~stack] [~elserj]
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIter
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381169#comment-16381169
]
Andrew Purtell commented on HBASE-19769:
So the decision was to (permanently?) revert HBASE-17448 here.
I don't see a notice on HBASE-17448 informing folks there this was going to
happen.
Since it is out from 2.0, should we revert it from 1.5 too?
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
> ... 21 more
> Caused by: java.lang.IllegalAccessError: tried to access class
> org.apache.hado
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324902#comment-16324902
]
stack commented on HBASE-19769:
---
[~elserj] RN? And this incompat change given it shipped in 1.4? Might want to
note it. Good on you Josh.
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
> ... 21 more
> Caused by: java.lang.IllegalAccessError: tried to access class
> org.apache.hadoop.metrics2.lib.MetricsInfoImpl from class
> org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry
> at
> org.ap
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324808#comment-16324808
]
Hudson commented on HBASE-19769:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4390 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/4390/])
HBASE-19769 Remove ZK metrics because of classloader issues (elserj: rev
057e80c1639412b25c2b3462851729a33cac39c7)
* (delete)
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKMetrics.java
* (delete)
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKMetricsListener.java
* (edit)
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
* (delete)
hbase-zookeeper/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMetrics.java
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextSer
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323416#comment-16323416
]
stack commented on HBASE-19769:
---
+1 then for master and branch-2.
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
> ... 21 more
> Caused by: java.lang.IllegalAccessError: tried to access class
> org.apache.hadoop.metrics2.lib.MetricsInfoImpl from class
> org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry
> at
> org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry.newGauge(DynamicMetri
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323246#comment-16323246
]
Josh Elser commented on HBASE-19769:
bq. Will put this on my test cluster to verify it works.
Rebuilt Phoenix against a version of HBase with this patch and the job went
through swimmingly.
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19769.001.branch-2.patch
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
> ... 21 more
> Caused by: java.lang.IllegalAccessError: tried to access class
> org.apache.hadoop.metrics2.lib.MetricsInfoImpl from class
> org.apache.hadoop
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323241#comment-16323241
]
Hadoop QA commented on HBASE-19769:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
27s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
44s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
28s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 3s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hbase-zookeeper in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19769 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905768/HBASE-19769.001.branch-2.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 272abd659189 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 3e6f80dcd5 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11031/testReport/ |
| modules | C: hbase-zookeeper U: hbase-zookeeper |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11031/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
>
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323152#comment-16323152
]
Josh Elser commented on HBASE-19769:
bq. Gotcha! I was still stuck on leaving them server-side. Removing them
outright is fine.
Let me make a patch for this. I had essentially done this already to verify
that this was the problem. After nulling out the ZKMetrics class, I was able to
run through Phoenix's CSVBulkLoad tool.
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
> ... 21 more
> Caused by: java.lang.IllegalAccessError: tried to access class
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323137#comment-16323137
]
Josh Elser commented on HBASE-19769:
bq. Not sure I follow. To be clear, strip all zk metrics. The thing is too
verbose anyways and can look at ensemble and hbase logs to figure whats going
on?
Gotcha! I was still stuck on leaving them server-side. Removing them outright
is fine.
I think for client-metrics, we should use the work Ron has put up in
HBASE-18409 as the path forward. That will help avoid this hitting us yet again
(I think).
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(Compati
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323128#comment-16323128
]
stack commented on HBASE-19769:
---
bq. The problem with #1 is that I'm not sure how we determine when we're in the
context of a user and when we're in the context of a service.
Not sure I follow. To be clear, strip all zk metrics. The thing is too verbose
anyways and can look at ensemble and hbase logs to figure whats going on?
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
> ... 21 more
> Caused by: java.lang.IllegalAccessError: tried to access class
> o
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323120#comment-16323120
]
Josh Elser commented on HBASE-19769:
bq. #1 sounds easy-to-do? Nice to have but an aspect of general client metrics
and user is not completely lost in that they can look at zk ensemble metrics.
#1 has never shipped in any hbase?
Sadly, it shipped in 1.4.0 -- makes me surprised how we didn't surface this in
1.4.0 testing. I guess we launch our Tools directly, and don't exec out to
yarn-jar or anything which would explain it. I haven't tested there yet.
The problem with #1 is that I'm not sure how we determine when we're in the
context of a user and when we're in the context of a service.
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>
[jira] [Commented] (HBASE-19769) IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs
[
https://issues.apache.org/jira/browse/HBASE-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323051#comment-16323051
]
stack commented on HBASE-19769:
---
We don't want to use hadoops metrics long-term if thats of any use to. Its only
been a pain. Modules for hadoop1 vs hadoop2. We have hacks in place to kill it
and restart it because no other way of onlining new metrics -- e.g. region
added/removed. We spend most of our CPU counting so we should probably own the
metrics system rather than defer to another.
#1 sounds easy-to-do? Nice to have but an aspect of general client metrics and
user is not completely lost in that they can look at zk ensemble metrics. #1
has never shipped in any hbase?
> IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce
> jobs
> ---
>
> Key: HBASE-19769
> URL: https://issues.apache.org/jira/browse/HBASE-19769
> Project: HBase
> Issue Type: Bug
> Components: mapreduce, metrics
>Affects Versions: 2.0.0-beta-1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0-beta-2
>
>
> issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893
> Since Hadoop 2.6.0, the {{yarn jar}} entry point to submit a YARN job has
> been using a custom classloader to separate Hadoop dependencies from the
> user's JAR being run. A separate classloader is created for the user-provided
> jar, and then this classloader is set as the contextClassLoader before the
> Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for
> us to date because we don't try to access any Hadoop internal classes
> client-side.
> However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The
> problem is that Hadoop metrics2 implementations which we reference from the
> same package are loaded by a different classloader than our HBase code is
> loaded from. This makes the expected package-private access of these Metrics2
> classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.
> {noformat}
> java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop
> compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKMetrics.(ZKMetrics.java:36)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:115)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:128)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.(ZKWatcher.java:102)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
> at
> org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
> at
> org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.util.ServiceConfigurationError:
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider
> org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.Se
