[ http://issues.apache.org/jira/browse/HADOOP-474?page=comments#action_12431056 ] Doug Cutting commented on HADOOP-474: -------------------------------------
> We do want the io.compression.codecs so that it is easy to extend the list of > potential codecs. You mean, so that its easy to extend the mapping from file extension to codec, right? Is there any other reason to enumerate codecs? > I would think that the SequenceFileOutputFormat should use the same property. But codec there will be noted inside the file, not in the extension, right? And compression there can either be value-only or of blocks of keys and values. So, even if you had a file extension, it wouldn't tell the whole story. So I don't think the property has anything to do with SequenceFile. So the property should proabably have 'text' in its name. > support compressed text files as input and output > ------------------------------------------------- > > Key: HADOOP-474 > URL: http://issues.apache.org/jira/browse/HADOOP-474 > Project: Hadoop > Issue Type: Improvement > Components: mapred > Affects Versions: 0.5.0 > Reporter: Owen O'Malley > Assigned To: Owen O'Malley > Fix For: 0.6.0 > > > I'd like TextInputFomat and TextOutputFormat to automatically compress and > uncompress text files when they are read and written. Furthermore, I'd like > to be able to use custom compressors as defined in HADOOP-441. Therefore, I > propose: > Adding a map of compression codecs in the server config files: > io.compression.codecs = "<suffix>=<codec class>,..." > so the default would be something like: > <property> > <name>io.compression.codecs</name> > > <value>.gz=org.apache.hadoop.io.GZipCodec,.Z=org.apache.hadoop.io.ZipCodec</value> > <description>A list of file suffixes and the codecs for them.</description> > </property> > note that the suffix can include multiple "." so you could support suffixes > like ".tar.gz", but they are just treated as literals against the end of the > filename. > If the TextInputFormat is dealing with such a file, it: > 1. makes a single split > 2. decompresses automatically > On the output side, if mapred.output.compress is true, then TextOutputFormat > would use a new property mapred.output.compression.codec that would define > the codec to use to compress the outputs, defaulting to gzip. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
