Hey, I feel HashSet is a good method to dedup. To increase the overall efficiency you could also look into Combiner running the same Reducer code. That would ensure less data in the sort-shuffle phase.
Regards, Matthew On Wed, Aug 3, 2011 at 11:52 AM, Jianxin Wang <wangjx...@gmail.com> wrote: > hi,harsh > After map, I can get all values for one key, but I want dedup these > values, only get all unique values. now I just do it like the image. > > I think the following code is not efficient.(using a HashSet to dedup) > Thanks:) > > private static class MyReducer extends > Reducer<LongWritable,LongWritable,LongWritable,LongsWritable> > { > HashSet<Long> uids=new HashSet<Long>(); > LongsWritable unique_uids=new LongsWritable(); > public void reduce(LongWritable key,Iterable<LongWritable> values,Context > context)throws IOException,InterruptedException > { > uids.clear(); > for(LongWritable v:values) > { > uids.add(v.get()); > } > int size=uids.size(); > long[] l=new long[size]; > int i=0; > for(long uid:uids) > { > l[i]=uid; > i++; > } > unique_uids.Set(l); > context.write(key,unique_uids); > } > } > > > 2011/8/3 Harsh J <ha...@cloudera.com> > >> Use MapReduce :) >> >> If map output: (key, value) >> Then reduce input becomes: (key, [iterator of values across all maps >> with (key, value)]) >> >> I believe this is very similar to the wordcount example, but minus the >> summing. For a given key, you get all the values that carry that key >> in the reducer. Have you tried to run a simple program to achieve this >> before asking? Or is something specifically not working? >> >> On Wed, Aug 3, 2011 at 9:20 AM, Jianxin Wang <wangjx...@gmail.com> wrote: >> > HI, >> > I hava many <key,value> pairs now, and want to get all different >> values >> > for each key, which way is efficient for this work. >> > >> > such as input : <1,2> <1,3> <1,4> <1,3> <2,1> <2,2> >> > output: <1,2/3/4> <2,1/2> >> > >> > Thanks! >> > >> > walter >> > >> >> >> >> -- >> Harsh J >> > >