Hi, It looks a problem of class path at spark side.
Thanks, - Tsuyoshi On Fri, Aug 29, 2014 at 8:49 AM, arthur.hk.c...@gmail.com <arthur.hk.c...@gmail.com> wrote: > Hi, > > I use Hadoop 2.4.1, I got "org.apache.hadoop.io.compress.SnappyCodec not > found” error: > > hadoop checknative > 14/08/29 02:54:51 WARN bzip2.Bzip2Factory: Failed to load/initialize > native-bzip2 library system-native, will use pure-Java version > 14/08/29 02:54:51 INFO zlib.ZlibFactory: Successfully loaded & initialized > native-zlib library > Native library checking: > hadoop: true > /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libhadoop.so > zlib: true /lib64/libz.so.1 > snappy: true > /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libsnappy.so.1 > lz4: true revision:99 > bzip2: false > > (smoke test is ok) > bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar > teragen 300000 /tmp/teragenout > 14/08/29 07:40:41 INFO mapreduce.Job: Running job: job_1409253811850_0002 > 14/08/29 07:40:53 INFO mapreduce.Job: Job job_1409253811850_0002 running in > uber mode : false > 14/08/29 07:40:53 INFO mapreduce.Job: map 0% reduce 0% > 14/08/29 07:41:00 INFO mapreduce.Job: map 50% reduce 0% > 14/08/29 07:41:01 INFO mapreduce.Job: map 100% reduce 0% > 14/08/29 07:41:02 INFO mapreduce.Job: Job job_1409253811850_0002 completed > successfully > 14/08/29 07:41:02 INFO mapreduce.Job: Counters: 31 > File System Counters > FILE: Number of bytes read=0 > FILE: Number of bytes written=197312 > FILE: Number of read operations=0 > FILE: Number of large read operations=0 > FILE: Number of write operations=0 > HDFS: Number of bytes read=167 > HDFS: Number of bytes written=30000000 > HDFS: Number of read operations=8 > HDFS: Number of large read operations=0 > HDFS: Number of write operations=4 > Job Counters > Launched map tasks=2 > Other local map tasks=2 > Total time spent by all maps in occupied slots (ms)=11925 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=11925 > Total vcore-seconds taken by all map tasks=11925 > Total megabyte-seconds taken by all map tasks=109900800 > Map-Reduce Framework > Map input records=300000 > Map output records=300000 > Input split bytes=167 > Spilled Records=0 > Failed Shuffles=0 > Merged Map outputs=0 > GC time elapsed (ms)=22 > CPU time spent (ms)=1910 > Physical memory (bytes) snapshot=357318656 > Virtual memory (bytes) snapshot=1691631616 > Total committed heap usage (bytes)=401997824 > org.apache.hadoop.examples.terasort.TeraGen$Counters > CHECKSUM=644086318705578 > File Input Format Counters > Bytes Read=0 > File Output Format Counters > Bytes Written=30000000 > 14/08/29 07:41:03 INFO terasort.TeraSort: starting > 14/08/29 07:41:03 INFO input.FileInputFormat: Total input paths to process : > 2 > > > However I got "org.apache.hadoop.io.compress.SnappyCodec not found” when > running spark smoke test program: > > scala> inFILE.first() > java.lang.RuntimeException: Error in configuring object > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) > at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) > at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158) > at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.rdd.RDD.partitions(RDD.scala:202) > at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.rdd.RDD.partitions(RDD.scala:202) > at org.apache.spark.rdd.RDD.take(RDD.scala:983) > at org.apache.spark.rdd.RDD.first(RDD.scala:1015) > at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15) > at $iwC$$iwC$$iwC.<init>(<console>:20) > at $iwC$$iwC.<init>(<console>:22) > at $iwC.<init>(<console>:24) > at <init>(<console>:26) > at .<init>(<console>:30) > at .<clinit>(<console>) > at .<init>(<console>:7) > at .<clinit>(<console>) > at $print(<console>) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788) > at > org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056) > at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614) > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645) > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609) > at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796) > at > org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841) > at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753) > at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601) > at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608) > at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611) > at > org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936) > at > org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884) > at > org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884) > at > scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) > at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884) > at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982) > at org.apache.spark.repl.Main$.main(Main.scala:31) > at org.apache.spark.repl.Main.main(Main.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) > ... 55 more > Caused by: java.lang.IllegalArgumentException: Compression codec > org.apache.hadoop.io.compress.SnappyCodec not found. > at > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135) > at > org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175) > at > org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45) > ... 60 more > Caused by: java.lang.ClassNotFoundException: Class > org.apache.hadoop.io.compress.SnappyCodec not found > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) > at > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128) > ... 62 more > > > > This is my setting in mapred-site.xml related to snappy > <property> > <name>mapred.output.compress</name> > <value>false</value> > <description>Should the job outputs be compressed?</description> > </property> > <property> > <name>mapred.output.compression.type</name> > <value>RECORD</value> > <description>If the job outputs are to compressed as SequenceFiles, how > should they be compressed? Should be one of NONE, RECORD or > BLOCK.</description> > </property> > <property> > <name>mapred.output.compression.codec</name> > <value>org.apache.hadoop.io.compress.SnappyCodec</value> > <description>If the job outputs are compressed, how should they be > compressed? > </description> > </property> > <property> > <name>mapred.compress.map.output</name> > <value>true</value> > <description>Should the outputs of the maps be compressed before being > sent across the network. Uses SequenceFile compression.</description> > </property> > <property> > <name>mapred.map.output.compression.codec</name> > <value>org.apache.hadoop.io.compress.SnappyCodec</value> > <description>If the map outputs are compressed, how should they be > compressed?</description> > </property> > > <property> > <name>mapreduce.map.output.compress</name> > <value>true</value> > </property> > <property> > <name>mapred.map.output.compress.codec</name> > <value>org.apache.hadoop.io.compress.SnappyCodec</value> > </property> > > > > Can you please advise if my mapred-site.xml setting is incorrect? > > Regards > Arthur > -- - Tsuyoshi
