Hadoop has integrated snappy via installed native libraries instead of snappy-java.jar (ref https://issues.apache.org/jira/browse/HADOOP-7206) - You need to have the snappy system libraries (snappy and snappy-devel) installed before you compile hadoop. (RPMs are available on the web, http://pkgs.org/centos-5-rhel-5/epel-i386/21/ for example) - When you build hadoop, you will need to compile the native libraries(by passing -Dcompile.native=true to ant) to avail snappy support. - You also need to make sure that snappy system library is available on the library path for all mapreduce tasks at runtime. Usually if you install them on /usr/lib or /usr/local/lib, it should work.
HTH, +Vinod On Apr 14, 2012, at 4:36 AM, Bas Hickendorff wrote: > Hello, > > When I start a map-reduce job, it starts, and after a short while, > fails with the error below (SnappyCodec not found). > > I am currently starting the job from other Java code (so the Hadoop > executable in the bin directory is not used anymore), but in principle > this seems to work (in the admin of the Jobtracker the job shows up > when it starts). However after a short while the map task fails with: > > > java.lang.IllegalArgumentException: Compression codec > org.apache.hadoop.io.compress.SnappyCodec not found. > at > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96) > at > org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:62) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:522) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.io.compress.SnappyCodec > at java.net.URLClassLoader$1.run(URLClassLoader.java:217) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:205) > at java.lang.ClassLoader.loadClass(ClassLoader.java:321) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) > at java.lang.ClassLoader.loadClass(ClassLoader.java:266) > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:334) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820) > at > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:89) > ... 10 more > > > I confirmed that the SnappyCodec class is present in the > hadoop-core-1.0.2.jar, and the snappy-java-1.0.4.1.jar is present as > well. The directory of those jars is on the HADOOP_CLASSPATH, but it > seems it still cannot find it. I also checked that the config files of > Hadoop are read. I run all nodes on localhost. > > Any suggestions on what could be the cause of the issue? > > Regards, > > Bas
