Native LZO support was removed from Hadoop due to licensing restrictions. See http://www.cloudera.com/blog/2009/06/24/parallel-lzo-splittable-compression-for-hadoop/ for a writeup on how to reenable it in your local build.
- Aaron On Tue, Jul 21, 2009 at 7:02 PM, Gross, Danny<danny.gr...@spansion.com> wrote: > Hello, > > > > I've been running terasort on multiple cluster configurations, and > attempted to duplicate some of the configuration settings that Yahoo! > used for the Minute Sort. > > > > In particular, I set the mapred.map.output.compression.codec property to > value "org.apache.hadoop.io.compress.LzoCodec" in hadoop-site.xml. I > am using hadoop-0.19.1. > > > > The teragen program runs fine, and completes with improved time with my > new settings. However, when I run the terasort program, the following > error is thrown from the map tasks, and the job ultimately fails: > > > > "java.lang.RuntimeException: native-lzo library not available at > org.apache.hadoop.io.compress.LzoCodec.getCompressorType(LzoCodec.java:1 > 30) at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:98) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:93) at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.ja > va:961) at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:842) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at > org.apache.hadoop.mapred.Child.main(Child.java:158)" > > > > I've searched other places for an answer, and am coming up short. Any > help out there would be greatly appreciated. > > > > Best regards, > > > > Danny > > > > Danny B. Gross > > Solutions Engineering > > Spansion, Inc. > > email: danny.gr...@spansion.com <mailto:danny.gr...@spansion.com> > > > >
