This works for us SET hive.exec.compress.intermediate=true SET hive.exec.compress.output=true SET mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec SET mapreduce.map.output.compress=true SET mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec SET mapreduce.output.fileoutputformat.compress=true
From: "bejoy...@yahoo.com<mailto:bejoy...@yahoo.com>" <bejoy...@yahoo.com<mailto:bejoy...@yahoo.com>> Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>, "bejoy...@yahoo.com<mailto:bejoy...@yahoo.com>" <bejoy...@yahoo.com<mailto:bejoy...@yahoo.com>> Date: Monday, July 22, 2013 5:09 AM To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>> Subject: Re: how to let hive support lzo Hi, Along with the mapred.compress* properties try to set hive.exec.compress.output to true. Regards Bejoy KS Sent from remote device, Please excuse typos ________________________________ From: ch huang <justlo...@gmail.com<mailto:justlo...@gmail.com>> Date: Mon, 22 Jul 2013 13:41:01 +0800 To: <user@hive.apache.org<mailto:user@hive.apache.org>> ReplyTo: user@hive.apache.org<mailto:user@hive.apache.org> Subject: Re: how to let hive support lzo # hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://CH22:9000/alex/my.txt lzo 13/07/22 13:27:58 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available 13/07/22 13:27:59 INFO util.ChecksumType: Checksum using org.apache.hadoop.util.PureJavaCrc32 13/07/22 13:27:59 INFO util.ChecksumType: Checksum can use org.apache.hadoop.util.PureJavaCrc32C 13/07/22 13:27:59 ERROR metrics.SchemaMetrics: Inconsistent configuration. Previous configuration for using table name in metrics: true, new configuration: false 13/07/22 13:27:59 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 13/07/22 13:27:59 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8] 13/07/22 13:27:59 INFO compress.CodecPool: Got brand-new compressor [.lzo_deflate] 13/07/22 13:28:00 INFO compress.CodecPool: Got brand-new decompressor [.lzo_deflate] SUCCESS # hadoop jar /usr/lib/hadoop/lib/hadoop-lzo-0.4.15.jar com.hadoop.compression.lzo.LzoIndexer /alex 13/07/22 09:39:04 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 13/07/22 09:39:04 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8] 13/07/22 09:39:04 INFO lzo.LzoIndexer: LZO Indexing directory /alex... 13/07/22 09:39:04 INFO lzo.LzoIndexer: LZO Indexing directory hdfs://CH22:9000/alex/alex_t... 13/07/22 09:39:04 INFO lzo.LzoIndexer: [INDEX] LZO Indexing file hdfs://CH22:9000/alex/sqoop-1.99.2-bin-hadoop200.tar.gz.lzo, size 0.02 GB... 13/07/22 09:39:05 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available 13/07/22 09:39:06 INFO lzo.LzoIndexer: Completed LZO Indexing in 1.16 seconds (13.99 MB/s). Index size is 0.52 KB. 13/07/22 09:39:06 INFO lzo.LzoIndexer: [INDEX] LZO Indexing file hdfs://CH22:9000/alex/test1.lzo, size 0.00 GB... 13/07/22 09:39:06 INFO lzo.LzoIndexer: Completed LZO Indexing in 0.08 seconds (0.00 MB/s). Index size is 0.01 KB. On Mon, Jul 22, 2013 at 1:37 PM, ch huang <justlo...@gmail.com<mailto:justlo...@gmail.com>> wrote: hi ,all: i already install and testing lzo in hadoop and hbase,all success,but when i try it on hive ,it failed ,how can i do let hive can recognize lzo? hive> set mapred.map.output.compression.codec; mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec hive> set mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzoCodec hive> select count(*) from test; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapred.reduce.tasks=<number> Starting Job = job_1374463239553_0003, Tracking URL = http://CH22:8088/proxy/application_1374463239553_0003/<http://ch22:8088/proxy/application_1374463239553_0003/> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1374463239553_0003 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2013-07-22 13:33:27,243 Stage-1 map = 0%, reduce = 0% 2013-07-22 13:33:45,403 Stage-1 map = 100%, reduce = 0% Ended Job = job_1374463239553_0003 with errors Error during job, obtaining debugging information... Job Tracking URL: http://CH22:8088/proxy/application_1374463239553_0003/<http://ch22:8088/proxy/application_1374463239553_0003/> Examining task ID: task_1374463239553_0003_m_000000 (and more) from job job_1374463239553_0003 Task with the most failures(4): ----- Task ID: task_1374463239553_0003_m_000000 URL: http://CH22:8088/taskdetails.jsp?jobid=job_1374463239553_0003&tipid=task_1374463239553_0003_m_000000<http://ch22:8088/taskdetails.jsp?jobid=job_1374463239553_0003&tipid=task_1374463239553_0003_m_000000> ----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: native-lzo library not available at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:104) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:118) at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:115) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1580) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1457) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) CONFIDENTIALITY NOTICE ====================== This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
