Hi Yong, A few ways to understand a problem (with Hadoop) : 1) Write tests, using MRUnit for example (http://mrunit.apache.org/) 2) Use @Override to make sure you are indeed overriding a method 3) Use a break point while debugging
The answer to your current problem : o.a.h.mapreduce.Reducer method has no Iterator parameter but it does have a Iterable parameter... Regards Bertrand PS : It is absolutely not related to HBase. On Thu, Nov 15, 2012 at 8:42 PM, yonghu <yongyong...@gmail.com> wrote: > Dear all, > > I use HBase as data source and hdfs as data sink. I wrote the program > which calculate the versions for each cell as follows: > > public class ReturnMore { //only Map side has no problem, has already > returned 3 versions > > public static class Map extends TableMapper<Text, Text>{ > private Text outkey = new Text(); > private Text outval = new Text(); > public void map(ImmutableBytesWritable key, Result values, Context > context){ > KeyValue[] kvs = values.raw(); > String row_key; > String col_fam; > String col; > String val; > String finalKey; > String finalValue; > for(KeyValue kv : kvs){ > row_key = Bytes.toString(kv.getRow()); > col_fam = Bytes.toString(kv.getFamily()); > col = Bytes.toString(kv.getQualifier()); > val = Bytes.toString(kv.getValue()); > finalKey = row_key + "/" + col_fam + "/" + col; > finalValue = new Long(kv.getTimestamp()).toString() + "/" + > val; > outkey.set(finalKey); > outval.set(finalValue); > try { > context.write(outkey, outval); > } catch (IOException e) { > e.printStackTrace(); > } catch (InterruptedException e) { > e.printStackTrace(); > } > } > } > } > > public static class Reduce extends Reducer<Text, Text, Text, > IntWritable>{ //the desired output is key + 3 > private IntWritable occ = new IntWritable(); > public void reduce(Text key, Iterator<Text> values, Context > context){ > int i = 0; > while(values.hasNext()){ > i++; > } > occ.set(i); > try { > context.write(key, occ); > } catch (IOException e) { > e.printStackTrace(); > } catch (InterruptedException e) { > e.printStackTrace(); > } > } > } > public static void main(String[] args) throws Exception{ > Configuration conf = HBaseConfiguration.create(); > Job job = new Job(conf); > job.setJarByClass(ReturnMore.class); > Scan scan = new Scan(); > scan.setMaxVersions(); > job.setReducerClass(Reduce.class); > job.setOutputKeyClass(Text.class); > job.setOutputValueClass(IntWritable.class); > job.setNumReduceTasks(1); > TableMapReduceUtil.initTableMapperJob("test", scan, Map.class, > Text.class, Text.class, job); > FileSystem file = FileSystem.get(conf); > Path path = new Path("/hans/test/"); > if(file.exists(path)){ > file.delete(path,true); > } > FileOutputFormat.setOutputPath(job, path); > System.exit(job.waitForCompletion(true)?0:1); > } > } > > I tested this code in both hbase 0.92.1 and 0.94. However, if I run > this code, it always outputs the content for each cell not as the > output as I defined in reduce function (key + occurrence for each > cell). Can anyone give me advices? By the way, I run it on > pseudo-mode. > > regards! > > Yong > -- Bertrand Dechoux