I tried conf.setBoolean("mapred.compress.map.output", true); but it didn't seem
to work.
Also, since I'm using the Avro mapred APIs, maybe there's something Avro
specific to get it enabled?
Should I ask on the Avro mailing lists?
Thanks,
Frank Grimes
On 2012-01-12, at 10:49 AM, bejoy.had...@gmail.com wrote:
> Hi Frank
> Is map output compression enabled?
>
> The config param would be like
> mapred.map.output.compress=true
> (It is from my memory, Please cross check)
>
> ------Original Message------
> From: Frank Grimes
> To: hdfs-user@hadoop.apache.org
> ReplyTo: hdfs-user@hadoop.apache.org
> Subject: Can spill to disk be in compressed format to reduce I/O?
> Sent: Jan 12, 2012 21:10
>
> Hi All,
>
> We're trying to speed up an M/R job which combines multiple .avro files.
> We've noticed that when it spills to disk, it's in uncompressed format.
> Is there a way to make it spill temporary segments as .avro with Deflate
> compression?
>
> Thanks,
>
> Frank Grimes
>
> Regards
> Bejoy K S
