One way of doing what you need is to extend MultipleTextOutputFormat and
override the following APIs
- generateFileNameForKeyValue()
- generateActualKey()
- generateActualValue()
You will need to prefix the directory and file-name of your choice to the
key/value depending upon your needs. Assuming key and value types to be Text
here is some sample code for reference
public String generateFileNameForKeyValue(Text key, Text v, String name) {
/*
* split the default name (for e.x. part-00000 into ['part', '00000'] )
*/
String[] nameparts = name.split("-");
String keyStr = key.toString();
/**
* assuming desired filename is prefixed to the key and separated from the
* actual key contents by '\t'
*/
int idx = keyStr.indexOf("\t");
/*
* get the file name
*/
name = keyStr.substring(0, idx);
/**
* return the path of the form 'fileName/fileName-0000'
* This makes sure that fileName dir is created under job's output dir
* and all the keys with that prefix go into reducer-specific files under
* that dir.
*/
return new Path(name, name + "-" + nameparts[1]).toString();
}
public Text generateActualKey(Text key, Text value) {
String keyStr = key.toString();
int idx = keyStr.indexOf("\t") + 1;
return new Text(keyStr.substring(idx));
}
Hope that helps.
-Ankur
----- Original Message -----
From: "Kevin Peterson" <kpeter...@biz360.com>
To: core-user@hadoop.apache.org
Sent: Friday, May 29, 2009 4:55:22 AM GMT +05:30 Chennai, Kolkata, Mumbai, New
Delhi
Subject: MultipleOutputs or MultipleTextOutputFormat?
I am trying to figure out the best way to split output into different
directories. My goal is to have a directory structure allowing me to add the
content from each batch into the right bucket, like this:
...
/content/200904/batch_20090429
/content/200904/batch_20090430
/content/200904/batch_20090501
/content/200904/batch_20090502
/content/200905/batch_20090430
/content/200905/batch_20090501
/content/200905/batch_20090502
...
I would then run my nightly jobs to build the index on /content/200904/* for
the April index and /content/200905/* for the May index.
I'm not sure whether I would be better off using MultipleOutputs or
MultipleTextOutputFormat. I'm having trouble understanding how I set the
output path for these two classes. It seems like MultipleTextOutputFormat is
about partitioning data to different files within the same directory on the
key, rather than into different directories as I need. Could I get the
behavior I want by specifying date/batch as my filename, set output path to
some temporary work directory, then move /work/* to /content?
MultipleOutputs seems to be more about outputting all the data in different
formats, but it's supposed to be simpler to use. Reading it, it seems to be
better documented and the API makes more sense (choosing the output
explicitly in the map or reduce, rather than hiding this decision in the
output format), but I don't see any way to set a file name. If am using
textoutputformat, I see no way to put these into different directories.
