Hi James, Thank you very much! :-)
-SM On Fri, Jan 30, 2009 at 4:17 PM, james warren <[email protected]> wrote: > Hello Sandy - > Your partitioner isn't using any information from the key/value pair - it's > only using the value T which is read once from the job configuration. > getPartition() will always return the same value, so all of your data is > being sent to one reducer. :P > > cheers, > -James > > On Fri, Jan 30, 2009 at 1:32 PM, Sandy <[email protected]> wrote: > > > Hello, > > > > Could someone point me toward some more documentation on how to write > one's > > own partition class? I have having quite a bit of trouble getting mine to > > work. So far, it looks something like this: > > > > public class myPartitioner extends MapReduceBase implements > > Partitioner<IntWritable, IntWritable> { > > > > private int T; > > > > public void configure(JobConf job) { > > super.configure(job); > > String myT = job.get("tval"); //this is user defined > > T = Integer.parseInt(myT); > > } > > > > public int getPartition(IntWritable key, IntWritable value, int > > numReduceTasks) { > > int newT = (T/numReduceTasks); > > int id = ((value.get()/ T); > > return (int)(id/newT); > > } > > } > > > > In the run() function of my M/R program I just set it using: > > > > conf.setPartitionerClass(myPartitioner.class); > > > > Is there anything else I need to set in the run() function? > > > > > > The code compiles fine. When I run it, I know it is "using" the > > partitioner, > > since I get different output than if I just let it use HashPartitioner. > > However, it is not splitting between the reducers at all! If I set the > > number of reducers to 2, all the output shows up in part-00000, while > > part-00001 has nothing. > > > > I am having trouble debugging this since I don't know how I can observe > the > > values of numReduceTasks (which I assume is being set by the system). Is > > this a proper assumption? > > > > If I try to insert any println() statements in the function, it isn't > > outputted to either my terminal or my log files. Could someone give me > some > > general advice on how best to debug pieces of code like this? > > >
