On 31 October 2011 14:13, Mathias Herberts wrote:
> Not one I can think of at the moment, but intuitively that's the kind of
> flexibility I would like to see should a grouping comparator become
> configurable for Combiners.
>
I don't think there's a good example, be
Not one I can think of at the moment, but intuitively that's the kind of
flexibility I would like to see should a grouping comparator become
configurable for Combiners.
By pushing your reasoning further, why specify a combiner class at all
instead of applying the Reducer on the map side.
O
sh J" wrote:
>
Now I'm curious. Can you argue that there's a case where it makes a
difference? Preferably one where it can't be trivially curried into the
combiner?
S.
> > Shevek,
> >
> > The problem Mathias indicates here is that the Combiners do not uti
I don't know if it's a bug but I'd rather have the ability to set a
Combiner specific group comparator than to have the Combiner use the group
comparator set for the Reducer.
On Oct 31, 2011 9:21 PM, "Harsh J" wrote:
> Shevek,
>
> The problem Mathias indicates
On 31 October 2011 13:20, Harsh J wrote:
> Shevek,
>
> The problem Mathias indicates here is that the Combiners do not utilize
> the Grouping Comparators. They only use the Sort Comparators. Is that
> probably a bug is what I wonder.
>
Hm, that's a rather interestin
Shevek,
The problem Mathias indicates here is that the Combiners do not utilize the
Grouping Comparators. They only use the Sort Comparators. Is that probably a
bug is what I wonder.
On 31-Oct-2011, at 11:14 PM, Shevek wrote:
> I like the ability to reuse a Java component for both sorting
ComparatorClass() API.
S.
On 29 October 2011 04:35, Mathias Herberts wrote:
> Another point concerning the Combiners,
>
> the grouping is currently done using the RawComparator used for
> sorting the Mapper's output. Wouldn't it be useful to be able to set a
> custom CombinerGroupingComparatorClass?
>
> Mathias.
>
d one option would be to
> ignore the output key.
>
>
> > Since a Combiner is simply a Reducer with no other constraints,
> >
>
> That isn't true. Combiners are required to be:
> 1. Idempotent - The number of times the combiner is applied can't change
> th
On Mon, Oct 31, 2011 at 5:41 AM, Mathias Herberts <
mathias.herbe...@gmail.com> wrote:
> Thanks for listing the 5 requirements, if you don't mind I'll add them
> to the Hadoop MapReducer Poster.
>
Sure.
> Yes. We've talked about adding various checks, but I don't think anyone has
> added them. We obviously have the input key and one option would be to
> ignore the output key.
ok.
>
>> Since a Combiner is simply a Reducer with no other constraints,
>>
>
>
It might end up in the wrong partition
>
Yes. We've talked about adding various checks, but I don't think anyone has
added them. We obviously have the input key and one option would be to
ignore the output key.
> Since a Combiner is simply a Reducer with no other constraints,
>
Another point concerning the Combiners,
the grouping is currently done using the RawComparator used for
sorting the Mapper's output. Wouldn't it be useful to be able to set a
custom CombinerGroupingComparatorClass?
Mathias.
Hi,
I'm designing a 'Hadoop MapReduce Poster', putting all pieces together
so people will easily be able to visualize the full M/R flow.
Concerning the combiners, I have a few points I'd like to have clarified.
If I'm not mistaken, the output of the Mapper is passed
13 matches
Mail list logo
