[
https://issues.apache.org/jira/browse/MAPREDUCE-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851171#action_12851171
]
Aaron Kimball commented on MAPREDUCE-1570:
------------------------------------------
Looks pretty good. The test failure is unrelated.
Some comments on the implementation in
src/contrib/mrunit/src/java/org/apache/hadoop/mrunit/MapReduceDriverBase.java:
{code}
+ keyValueOrderComparator = simpleComparator;
{code}
Are these ordinarily ordered by default? I thought they were placed in an
undefined ordering. Does this change the semantics of the programs under test
to enforce here? I think that the keyValueOrderComparator should remain null,
and in in shuffle() step 2, if this comparator is null, the secondary sort is
just skipped.
{code}
+ groupedKeyList = new Vector<Pair<K2,V2>>();
{code}
Don't use Vector; use ArrayList instead unless you specifically need thread
safety (I don't think you do here).
> Shuffle stage - Key and Group Comparators
> -----------------------------------------
>
> Key: MAPREDUCE-1570
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1570
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/mrunit
> Reporter: Chris White
> Assignee: Chris White
> Priority: Minor
> Attachments: MAPREDUCE-1570.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Shuffle method in org.apache.hadoop.mrunit.MapReduceDriverBase doesn't
> currently allow the use of custom GroupingComparator and SortComparator.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
