Hello Folks, I needed some help with using MultipleTextOutputFormat to control the output filename in MapReduce.
Currently I am using it as shown below(or at http://pastebin.com/gJxkdwRd). And it seems to work fine. However what I am trying to change is the usage of the fields that get picked to determine the filename. Instead of hardcoding them to field[0] or field[3](as is the case in the sample), I would like to pick this up (in some dynamic fashion) from say, JobConf as field[jobConf.get("id.offset")] or field[jobConf[get("date.offset")]. Does anyone here know how I could go about doing this (or something to this effect i.e. it doesn't have to be JobConf per se)? Any pointers/suggestions/tips et al. would be most appreciated. Thanks. PS: 1) Current usage of MultipleTextOutputFormat: public class FooBarMultipleTextOutputFormat extends MultipleTextOutputFormat<NullWritable, Text> { protected String generateFileNameForKeyValue(NullWritable key, Text value, String name) { String line = value.toString(); //TODO: I would like to parameterize the field that is picked //here. Something akin to using JobConf in Mapper. //i.e. instead of hard-coding [3] or [0], I would like //to get it from JobConf(or some other configuration) in some fashion String date = (line.split("\t"))[3].substring(0,10); String id = (line.split("\t"))[0]; String partitionNumber = String.format("%05d", ID.getPartitionNumber(id)); return date + "/pn_" + partitionNumber; } } 2) This is also an SO question here: http://goo.gl/FX7QN, if you would rather answer it there. TIA.
