[ https://issues.apache.org/jira/browse/HIVE-759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao resolved HIVE-759. ----------------------------- Resolution: Fixed Fix Version/s: 0.5.0 Release Note: HIVE-759. Add "hive.intermediate.compression.codec/type" option. (Yongqiang He via zshao) Hadoop Flags: [Reviewed] Committed. Thanks Yongqiang. > add hive.intermediate.compression.codec option > ---------------------------------------------- > > Key: HIVE-759 > URL: https://issues.apache.org/jira/browse/HIVE-759 > Project: Hadoop Hive > Issue Type: Improvement > Components: Query Processor > Reporter: Zheng Shao > Assignee: He Yongqiang > Fix For: 0.5.0 > > Attachments: hive-759-2009-08-17.patch, hive-759-2009-08-18-2.patch, > hive-759-2009-08-18.patch > > > Hive uses the jobconf compression codec for all map-reduce jobs. This > includes both mapred.map.output.compression.codec and > mapred.output.compression.codec. > In some cases, we want to distinguish between the codec used for intermediate > map-reduce jobs (that produces intermediate data between jobs) and the final > map-reduce jobs (that produces data stored in tables). > For intermediate data, lzo might be a better fit because it's much faster; > for final data, gzip might be a better fit because it saves disk spaces. > We should introduce two new options: > {code} > hive.intermediate.compression.codec=org.apache.hadoop.io.compress.LzoCodec > hive.intermediate.compression.type=BLOCK > {code} > And use these 2 options to override the mapred.output.compression.* in the > FileSinkOperator that produces intermediate data. > Note that it's possible that a single map-reduce job may have 2 > FileSInkOperators: one produces intermediate data, and one produces final > data. So we need to add a flag to fileSinkDesc for that. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.