[
https://issues.apache.org/jira/browse/MAPREDUCE-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799524#action_12799524
]
Chris Douglas commented on MAPREDUCE-1372:
------------------------------------------
bq. Maybe I misunderstood Arun's idea, but it sounded to me like it added a lot
of complexity. This JIRA indicates this bug exists in 0.20.1 - do we anticipate
fixing it for branch-20, or only for 21? If for branch-20, I think the
COWArraySet is safer, no?
Arun's idea is more complex, but it also addresses related problems. Different
fixes for 0.20/0.21/0.22 may make sense. A COWArraySet probably isn't the right
data structure per the preceding, but another of the concurrent collections
would be fine.
bq. What about ConcurrentSkipListSet? It's logarithmic and has weakly
consistent iteration.
Sure, that'd work. Again, it's backed by a map, just as HashSet and the current
patch's pseudo-ConcurrentHashSet. ConcurrentHashMap doesn't mention its
contains/insertion complexity in the javadoc, but debating its superiority to
log\(n) seems pointless for a degenerate case of tens of thousands of elements.
Since the type doesn't leak, being consistent with it doesn't make much
difference, either.
> ConcurrentModificationException in JobInProgress
> ------------------------------------------------
>
> Key: MAPREDUCE-1372
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1372
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.1
> Reporter: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: M1372-0.patch
>
>
> We have seen the following ConcurrentModificationException in one of our
> clusters
> {noformat}
> java.io.IOException: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:828)
> at
> org.apache.hadoop.mapred.JobInProgress.findNewMapTask(JobInProgress.java:2018)
> at
> org.apache.hadoop.mapred.JobInProgress.obtainNewMapTask(JobInProgress.java:1077)
> at
> org.apache.hadoop.mapred.CapacityTaskScheduler$MapSchedulingMgr.obtainNewTask(CapacityTaskScheduler.java:796)
> at
> org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.getTaskFromQueue(CapacityTaskScheduler.java:589)
> at
> org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.assignTasks(CapacityTaskScheduler.java:677)
> at
> org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.access$500(CapacityTaskScheduler.java:348)
> at
> org.apache.hadoop.mapred.CapacityTaskScheduler.addMapTask(CapacityTaskScheduler.java:1397)
> at
> org.apache.hadoop.mapred.CapacityTaskScheduler.assignTasks(CapacityTaskScheduler.java:1349)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2976)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
