Hah - I should've read the code more closely. Completely agree with Aaron's
assessment, and will drink more coffee before writing emails. :)
Thanks for the correction.
On Tue, Feb 3, 2009 at 4:42 PM, Aaron Kimball <aa...@cloudera.com> wrote:
> er?
>
> It seems to be using value.get(). That having been said, you should really
> partition based on key, not on value. (I am not sure why, exactly, the
> value
> is provided to the getPartition() method.)
>
>
> Moreover, I think the problem is that you are using division ( / ) not
> modulus ( % ). Your code simplifies to: (value.get() / T) / (T /
> numPartitions) = value.get() * numPartitions / T^2.
>
> The contract of getPartition() is that it returns a value in [0,
> numPartitions). The division operators are not guaranteed to return
> anything
> in this range, but (foo % numPartitions) will always do the right thing. So
> it's probably just assigning everything to reduce partition 0.
> (Alternatively, it could be that value * numPartitions < T^2 for any values
> of T you're testing with, which means that integer division will return 0.)
>
> - Aaron
>
>
> On Fri, Jan 30, 2009 at 3:43 PM, Sandy <snickerdoodl...@gmail.com> wrote:
>
> > Hi James,
> >
> > Thank you very much! :-)
> >
> > -SM
> >
> > On Fri, Jan 30, 2009 at 4:17 PM, james warren <ja...@rockyou.com> wrote:
> >
> > > Hello Sandy -
> > > Your partitioner isn't using any information from the key/value pair -
> > it's
> > > only using the value T which is read once from the job configuration.
> > > getPartition() will always return the same value, so all of your data
> is
> > > being sent to one reducer. :P
> > >
> > > cheers,
> > > -James
> > >
> > > On Fri, Jan 30, 2009 at 1:32 PM, Sandy <snickerdoodl...@gmail.com>
> > wrote:
> > >
> > > > Hello,
> > > >
> > > > Could someone point me toward some more documentation on how to write
> > > one's
> > > > own partition class? I have having quite a bit of trouble getting
> mine
> > to
> > > > work. So far, it looks something like this:
> > > >
> > > > public class myPartitioner extends MapReduceBase implements
> > > > Partitioner<IntWritable, IntWritable> {
> > > >
> > > > private int T;
> > > >
> > > > public void configure(JobConf job) {
> > > > super.configure(job);
> > > > String myT = job.get("tval"); //this is user defined
> > > > T = Integer.parseInt(myT);
> > > > }
> > > >
> > > > public int getPartition(IntWritable key, IntWritable value, int
> > > > numReduceTasks) {
> > > > int newT = (T/numReduceTasks);
> > > > int id = ((value.get()/ T);
> > > > return (int)(id/newT);
> > > > }
> > > > }
> > > >
> > > > In the run() function of my M/R program I just set it using:
> > > >
> > > > conf.setPartitionerClass(myPartitioner.class);
> > > >
> > > > Is there anything else I need to set in the run() function?
> > > >
> > > >
> > > > The code compiles fine. When I run it, I know it is "using" the
> > > > partitioner,
> > > > since I get different output than if I just let it use
> HashPartitioner.
> > > > However, it is not splitting between the reducers at all! If I set
> the
> > > > number of reducers to 2, all the output shows up in part-00000, while
> > > > part-00001 has nothing.
> > > >
> > > > I am having trouble debugging this since I don't know how I can
> observe
> > > the
> > > > values of numReduceTasks (which I assume is being set by the system).
> > Is
> > > > this a proper assumption?
> > > >
> > > > If I try to insert any println() statements in the function, it isn't
> > > > outputted to either my terminal or my log files. Could someone give
> me
> > > some
> > > > general advice on how best to debug pieces of code like this?
> > > >
> > >
> >
>
