[
https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007153#comment-13007153
]
Owen O'Malley commented on HADOOP-7183:
---------------------------------------
I'm not very happy with this fix. The implicit contract is that all comparators
in the cache must be thread safe. To special case the WritableComparator case
because it isn't thread safe seems like the wrong direction. It seems far less
brittle to use a thread local buffer for the WritableComparator.
Have you actually verified that the shuffle's sorts actually do a get per a
thread?
> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
> Key: HADOOP-7183
> URL: https://issues.apache.org/jira/browse/HADOOP-7183
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Tom White
> Priority: Blocker
> Fix For: 0.20.3, 0.21.1, 0.22.0
>
> Attachments: HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed
> WritableComparator gets saved back into the static map. This is fine for
> stateless comparators, but some comparators have per-instance state, and thus
> this becomes thread-unsafe and causes errors in the shuffle where multiple
> threads are doing comparisons. An example of a Comparator with per-instance
> state is WritableComparator itself.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
