Does your compareTo() method test object pointer equality? If so, you could be getting burned by Hadoop reusing Writable objects.
-Joey On Aug 14, 2011 9:20 PM, "Stan Rosenberg" <srosenb...@proclivitysystems.com> wrote: > Hi Folks, > > After much poking around I am still unable to determine why I am seeing > 'reduce' being called twice with the "same" key. > Recall from my previous email that "sameness" is determined by 'compareTo' > of my custom key type. > > AFAIK, the default WritableComparator invokes 'compareTo' for any two keys > which are being ordered during sorting and merging. > Is it somehow possible that a bitwise comparator is used for the spilled map > output rather than the default WritableComparator? > > I am out of clues, short of studying the "shuffling" code. If anyone can > suggest some further debugging steps, don't be shy. :) > > Thanks!!! > > stan
