[
https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294522#comment-13294522
]
Dave Beech commented on MRUNIT-114:
-----------------------------------
Hi Jim - I'm working with the KeyFieldBasedComparator at the moment and I've
found a couple more issues you might need to consider.
1. Because the shuffle method in MapReduceDriverBase calls the key ordering
comparator using compare(Object,Object), it does not call the correct method on
classes which only override and do all the work in compare(byte[] b1, int s1,
int l1, byte[] b2, int s2, int l2), like KeyFieldBasedComparator.
It might be better to check whether the two keys are BinaryComparable, casting,
and calling this method if possible. Even if a user's comparator doesn't
override it, the default implementation of this in WritableComparator would
still be OK.
2. KeyFieldBasedComparator doesn't get configured properly even after the
change in 1344239, because ReflectionUtils.newInstance requires the provided
config to be a JobConf in order for it to call the configure method on a
JobConfigurable object. Of course, the trouble is that simply wrapping
getConfiguration as new JobConf(getConfiguration()) isn't a great thing to do
as MapReduceDriverBase is shared by the mapreduce api version...
> should support calling configure on sorting and grouping comparators if
> applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jacob Metcalf
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it
> intentional that the configuration is not applied to the sorting and grouping
> comparators?
> I am writing my own multiple input MR driver to test a reduce side join and
> had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
