Hello,
I've been running terasort on multiple cluster configurations, and attempted to duplicate some of the configuration settings that Yahoo! used for the Minute Sort. In particular, I set the mapred.map.output.compression.codec property to value "org.apache.hadoop.io.compress.LzoCodec" in hadoop-site.xml. I am using hadoop-0.19.1. The teragen program runs fine, and completes with improved time with my new settings. However, when I run the terasort program, the following error is thrown from the map tasks, and the job ultimately fails: "java.lang.RuntimeException: native-lzo library not available at org.apache.hadoop.io.compress.LzoCodec.getCompressorType(LzoCodec.java:1 30) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:98) at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:93) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.ja va:961) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:842) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.Child.main(Child.java:158)" I've searched other places for an answer, and am coming up short. Any help out there would be greatly appreciated. Best regards, Danny Danny B. Gross Solutions Engineering Spansion, Inc. email: danny.gr...@spansion.com <mailto:danny.gr...@spansion.com>
