[
https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3531:
-------------------------------------------
Attachment: MR-3531.txt
Adding in a patch to fix the issue (Just changed HashMap to ConcurrentHashMap).
I also added in a test, but I don't really like the test that much. It tries
to brute force reproduce the problem. On my box it takes about 1.8 sec to run
so it should not be too big of an issue right now.
However, when trying to run without the fix I have seen it reproduce the
problem, but I have also seen it essentially get into a living deadlock where
all of the cores are pegged at 100% utilization, and sometimes only a kill -9
stops it form running. I am fine with removing the test, because I don't think
it adds much value, but I am also fine with leaving it in.
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation
> in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in
> handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at
> org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
>
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
