[ https://issues.apache.org/jira/browse/HADOOP-15401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16483453#comment-16483453 ]
Yongjun Zhang commented on HADOOP-15401: ---------------------------------------- HI [~xiaochen], Based on the discussion you had with [~daryn], it seems more reasonable to leave this jira Open, instead of Fixed. If we can not reproduce, we can mark it "Cant Reproduce". There is still the culprit to identify. Thanks. > ConcurrentModificationException on Subject.getPrivateCredentials in UGI > constructor > ----------------------------------------------------------------------------------- > > Key: HADOOP-15401 > URL: https://issues.apache.org/jira/browse/HADOOP-15401 > Project: Hadoop Common > Issue Type: Bug > Components: security > Affects Versions: 3.1.0, 3.0.3 > Reporter: Xiao Chen > Priority: Major > > Seen a recent exception from KMS client provider as follows: > {noformat} > java.io.IOException: java.util.ConcurrentModificationException > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:488) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:287) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:283) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:123) > at > org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:283) > at > org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532) > at > org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927) > at > org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946) > at > org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316) > at > org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323) > Caused by: java.util.ConcurrentModificationException > at > java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) > at java.util.LinkedList$ListItr.next(LinkedList.java:888) > at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070) > at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399) > at javax.security.auth.Subject$ClassSet.<init>(Subject.java:1372) > at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767) > at > org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(KerberosUtil.java:267) > at > org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:715) > at > org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:701) > at > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:742) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:141) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:477) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:472) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at > org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:471) > ... 12 more > {noformat} > It looks like we have ran into a race modifying jdk Subject class' > privCredentials. > Found [https://bugs.openjdk.java.net/browse/JDK-4892913] but that jira was > created before Hadoop.... > [~daryn], any thoughts on this? > (We have not seen this in versions pre-3.0 yet, but it seems HADOOP-9747 > would make solve this exact exception because it removed the access in UGI > constructor) -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org