[
https://issues.apache.org/jira/browse/HDFS-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439077#comment-13439077
]
Jason Lowe commented on HDFS-3831:
----------------------------------
Recently we've hit a sporadic issue where the presence of the HDFS
test-sources.jar prevents a secure cluster from running properly. Here's an
example exception from a client trying to launch a sample MapReduce job:
{noformat}
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.submitApplication(ClientRMProtocolPBClientImpl.java:174)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:318)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:292)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:383)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1216)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1213)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1213)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1234)
at org.apache.hadoop.examples.RandomWriter.run(RandomWriter.java:283)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.examples.RandomWriter.main(RandomWriter.java:294)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: com.google.protobuf.ServiceException:
java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
at
org.apache.hadoop.tools.TestDelegationTokenFetcher$FakeRenewer.handleKind(TestDelegationTokenFetcher.java:70)
at org.apache.hadoop.security.token.Token.getRenewer(Token.java:286)
at org.apache.hadoop.security.token.Token.isManaged(Token.java:301)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:279)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:280)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.handle(RMAppManager.java:333)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:255)
at
org.apache.hadoop.yarn.api.impl.pb.service.ClientRMProtocolPBServiceImpl.submitApplication(ClientRMProtocolPBServiceImpl.java:141)
at
org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.callBlockingMethod(ClientRMProtocol.java:178)
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:353)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1528)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1524)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1522)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
... 16 more
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
at $Proxy6.submitApplication(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.submitApplication(ClientRMProtocolPBClientImpl.java:167)
... 25 more
Caused by: java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
at
org.apache.hadoop.tools.TestDelegationTokenFetcher$FakeRenewer.handleKind(TestDelegationTokenFetcher.java:70)
at org.apache.hadoop.security.token.Token.getRenewer(Token.java:286)
at org.apache.hadoop.security.token.Token.isManaged(Token.java:301)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addApplication(DelegationTokenRenewer.java:279)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:280)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.handle(RMAppManager.java:333)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:255)
at
org.apache.hadoop.yarn.api.impl.pb.service.ClientRMProtocolPBServiceImpl.submitApplication(ClientRMProtocolPBServiceImpl.java:141)
at
org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.callBlockingMethod(ClientRMProtocol.java:178)
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:353)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1528)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1524)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1522)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
... 16 more
at org.apache.hadoop.ipc.Client.call(Client.java:1088)
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
... 27 more
{noformat}
What appears to be happening is that the service loader is finding
{{TestDelegationTokenFetcher$FakeRenewer}} as a token renewer and attempts to
instantiate and call its {{handleKind}} to see if it can handle renewing an
HDFS token. However loading this class causes it to try to load Mockito which
isn't available and the submission is aborted.
So why is it even finding {{TestDelegationTokenFetcher$FakeRenewer}}? The HDFS
tests-sources.jar provides a file that registers it with the service loader.
If that jar is in the classpath then we could attempt to load it if we don't
find a suitable token renewer earlier in the classpath. In the dist tarball,
the HDFS test-sources.jar is located in the same directory as the main HDFS
jar, and hadoop-config.sh sets up the classpath via globbing of that directory.
Globbing in a classpath does not guarantee ordering of jars within the
classpath, so sometimes the tests-sources.jar is appearing before the main HDFS
jar and that breaks the cluster when it comes to renewing HDFS tokens.
Even if the ordering were fixed, this could end up breaking other kinds of
tokens like Hive if those token renewers appear later in the classpath than
HDFS. Seems like we should reorganize the dist tarball so that test and
test-sources jars are located in another directory that we can easily filter
out of the classpath in hadoop-config.sh. There's still the issue of
semi-random ordering of the classpath when we use globbing, but that's probably
another JIRA.
> Failure to renew tokens due to test-sources left in classpath
> -------------------------------------------------------------
>
> Key: HDFS-3831
> URL: https://issues.apache.org/jira/browse/HDFS-3831
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 0.23.3, 2.1.0-alpha
> Reporter: Jason Lowe
> Priority: Critical
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
