[
https://issues.apache.org/jira/browse/HADOOP-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451674#comment-16451674
]
Xiao Chen commented on HADOOP-15408:
------------------------------------
Hi [~shahrs87], comments below and a request in the end.
Comments:
bq. Identifier for both the tokens are the same
Correct, and that's exactly why we can do the trick at KMSCP to copy a new kind.
bq. 4 tests are failing in TestKMS because they are trying to decodeIdentifier
with kind kms-dt and Serviceloader is not able to find any Identifier which
corresponds to kms-dt kind.
Thanks for testing. The patch was to express the idea - seems it won't compile
on trunk. I have cleaned up compilation, and don't see any test failures.
Attaching a new patch and we can run pre-commit to see.
Back to the problem itself, which I think is another compatibility dimension as
Steve mentioned, is that when old jars and new jars are both present, we should
behave the same way as if only new jars or only old jars are present.
The specific issue, as you discovered in the description, is that service
loader will load things in accumulation. This means
{{org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSDelegationTokenIdentifier}}
from both jars, and
{{org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier}}
from the new jar. I think this would trigger [service
loader|https://docs.oracle.com/javase/6/docs/api/java/util/ServiceLoader.html]
to ignore duplicates, so in the order you provided, we'll end up with token
identifier from the old jar, and legacy token identifier from the new jar,
which are both kms-dt. It seems then in the job, kms-dt mapped to the legacy
token (new jar), which upon looking up the {{TOKEN_LEGACY_KIND}} field reached
the old jar's {{KMSDelegationToken}} class. (I'm not totally sure but this
seems to be the only explanation for the stacetrace)
So, in split.patch what I tried to do is to make sure:
# {{org.apache.hadoop.security.token.TokenIdentifier}} have different classes
between old jar and new jar
# new jar's legacy token do not collide with old classes, and everything is
self-contained (within {{KMSDelegationTokenLegacy.java}}).
I think this should make service loader happy, and no matter kms-dt at run time
maps to {{KMSDelegationTokenIdentifier}} in the old jar, or
{{KMSLegacyDelegationTokenIdentifier}} in the new jar, the identifier can be
initialized.
Request:
Sorry I won't have cycles this week to try reproduce and debug on a real spark
case. Since you have a failing Spark job already, do you think you can test
this with the failing spark use case, and verify whether this fixes the issue?
Would be great if [~arpitagarwal] could try on the Ranger case as well....
Thanks!
> HADOOP-14445 broke Spark.
> -------------------------
>
> Key: HADOOP-15408
> URL: https://issues.apache.org/jira/browse/HADOOP-15408
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.8.4
> Reporter: Rushabh S Shah
> Priority: Blocker
> Attachments: split.patch, split.prelim.patch
>
>
> Spark bundles hadoop related jars in their package.
> Spark expects backwards compatibility between minor versions.
> Their job failed after we deployed HADOOP-14445 in our test cluster.
> {noformat}
> 2018-04-20 21:09:53,245 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2018-04-20 21:09:53,273 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.util.ServiceConfigurationError:
> org.apache.hadoop.security.token.TokenIdentifier: Provider
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$
> KMSLegacyDelegationTokenIdentifier could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:117)
> at org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:138)
> at org.apache.hadoop.security.token.Token.identifierToString(Token.java:393)
> at org.apache.hadoop.security.token.Token.toString(Token.java:413)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:155)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1634)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
> Caused by: java.lang.NoSuchFieldError: TOKEN_LEGACY_KIND
> at
> org.apache.hadoop.crypto.key.kms.KMSDelegationToken$KMSLegacyDelegationTokenIdentifier.<init>(KMSDelegationToken.java:64)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.lang.Class.newInstance(Class.java:442)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 10 more
> 2018-04-20 21:09:53,278 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1
> {noformat}
> Their classpath looks like
> {{\{...:hadoop-common-pre-HADOOP-14445.jar:.....:hadoop-common-with-HADOOP-14445.jar:....\}}}
> This is because the container loaded {{KMSDelegationToken}} class from an
> older jar and {{KMSLegacyDelegationTokenIdentifier}} from new jar and it
> fails when {{KMSLegacyDelegationTokenIdentifier}} wants to read
> {{TOKEN_LEGACY_KIND}} from {{KMSDelegationToken}} which doesn't exist before.
> Cc [~xiaochen]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
