# hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://CH22:9000/alex/my.txt lzo 13/07/22 13:27:58 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available 13/07/22 13:27:59 INFO util.ChecksumType: Checksum using org.apache.hadoop.util.PureJavaCrc32 13/07/22 13:27:59 INFO util.ChecksumType: Checksum can use org.apache.hadoop.util.PureJavaCrc32C 13/07/22 13:27:59 ERROR metrics.SchemaMetrics: Inconsistent configuration. Previous configuration for using table name in metrics: true, new configuration: false 13/07/22 13:27:59 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 13/07/22 13:27:59 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8] 13/07/22 13:27:59 INFO compress.CodecPool: Got brand-new compressor [.lzo_deflate] 13/07/22 13:28:00 INFO compress.CodecPool: Got brand-new decompressor [.lzo_deflate] SUCCESS
# hadoop jar /usr/lib/hadoop/lib/hadoop-lzo-0.4.15.jar com.hadoop.compression.lzo.LzoIndexer /alex 13/07/22 09:39:04 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 13/07/22 09:39:04 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8] 13/07/22 09:39:04 INFO lzo.LzoIndexer: LZO Indexing directory /alex... 13/07/22 09:39:04 INFO lzo.LzoIndexer: LZO Indexing directory hdfs://CH22:9000/alex/alex_t... 13/07/22 09:39:04 INFO lzo.LzoIndexer: [INDEX] LZO Indexing file hdfs://CH22:9000/alex/sqoop-1.99.2-bin-hadoop200.tar.gz.lzo, size 0.02 GB... 13/07/22 09:39:05 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available 13/07/22 09:39:06 INFO lzo.LzoIndexer: Completed LZO Indexing in 1.16 seconds (13.99 MB/s). Index size is 0.52 KB. 13/07/22 09:39:06 INFO lzo.LzoIndexer: [INDEX] LZO Indexing file hdfs://CH22:9000/alex/test1.lzo, size 0.00 GB... 13/07/22 09:39:06 INFO lzo.LzoIndexer: Completed LZO Indexing in 0.08 seconds (0.00 MB/s). Index size is 0.01 KB. On Mon, Jul 22, 2013 at 1:37 PM, ch huang <justlo...@gmail.com> wrote: > hi ,all: > i already install and testing lzo in hadoop and hbase,all success,but > when i try it on hive ,it failed ,how can i do let hive can recognize lzo? > > > hive> set mapred.map.output.compression.codec; > > mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec > hive> set > mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzoCodec > hive> select count(*) from test; > Total MapReduce jobs = 1 > Launching Job 1 out of 1 > Number of reduce tasks determined at compile time: 1 > In order to change the average load for a reducer (in bytes): > set hive.exec.reducers.bytes.per.reducer=<number> > In order to limit the maximum number of reducers: > set hive.exec.reducers.max=<number> > In order to set a constant number of reducers: > set mapred.reduce.tasks=<number> > Starting Job = job_1374463239553_0003, Tracking URL = > http://CH22:8088/proxy/application_1374463239553_0003/<http://ch22:8088/proxy/application_1374463239553_0003/> > Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1374463239553_0003 > Hadoop job information for Stage-1: number of mappers: 1; number of > reducers: 1 > 2013-07-22 13:33:27,243 Stage-1 map = 0%, reduce = 0% > 2013-07-22 13:33:45,403 Stage-1 map = 100%, reduce = 0% > Ended Job = job_1374463239553_0003 with errors > Error during job, obtaining debugging information... > Job Tracking URL: > http://CH22:8088/proxy/application_1374463239553_0003/<http://ch22:8088/proxy/application_1374463239553_0003/> > Examining task ID: task_1374463239553_0003_m_000000 (and more) from job > job_1374463239553_0003 > Task with the most failures(4): > ----- > Task ID: > task_1374463239553_0003_m_000000 > URL: > > http://CH22:8088/taskdetails.jsp?jobid=job_1374463239553_0003&tipid=task_1374463239553_0003_m_000000<http://ch22:8088/taskdetails.jsp?jobid=job_1374463239553_0003&tipid=task_1374463239553_0003_m_000000> > ----- > Diagnostic Messages for this Task: > Error: java.lang.RuntimeException: native-lzo library not available > at > com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:104) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:118) > at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:115) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1580) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1457) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) > >
