sorry for typo, mahout, not mahou. sent from mobile On Jul 11, 2013 9:40 PM, "Azuryy Yu" <azury...@gmail.com> wrote:
> hi, > > put all mahou jars under hadoop_home/lib, then restart cluster. > On Jul 11, 2013 8:45 PM, "Margusja" <mar...@roo.ee> wrote: > >> Hi >> >> I have tow nodes: >> n1 (master, salve) and n2 (slave) >> >> after set up I ran wordcount example and it worked fine: >> [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-**examples-1.0.4.jar >> wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output >> 13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to >> process : 7 >> 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop >> library >> 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded >> 13/07/11 15:30:44 INFO mapred.JobClient: Running job: >> job_201307111355_0015 >> 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0% >> 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0% >> 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0% >> 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0% >> 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0% >> 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23% >> 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100% >> 13/07/11 15:31:32 INFO mapred.JobClient: Job complete: >> job_201307111355_0015 >> 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30 >> 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters >> 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce tasks=1 >> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576 >> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all >> reduces waiting after reserving slots (ms)=0 >> 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all maps >> waiting after reserving slots (ms)=0 >> 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map tasks=3 >> 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7 >> 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map tasks=4 >> 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992 >> 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format Counters >> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505 >> 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters >> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195 >> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820 >> 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993 >> 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505 >> 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format Counters >> 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001 >> 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework >> 13/07/11 15:31:32 INFO mapred.JobClient: Map output materialized >> bytes=3157469 >> 13/07/11 15:31:32 INFO mapred.JobClient: Map input records=137146 >> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle bytes=2904836 >> 13/07/11 15:31:32 INFO mapred.JobClient: Spilled Records=594764 >> 13/07/11 15:31:32 INFO mapred.JobClient: Map output bytes=11435849 >> 13/07/11 15:31:32 INFO mapred.JobClient: Total committed heap usage >> (bytes)=1128136704 >> 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent (ms)=18230 >> 13/07/11 15:31:32 INFO mapred.JobClient: Combine input records=1174991 >> 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819 >> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input records=218990 >> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input groups=128513 >> 13/07/11 15:31:32 INFO mapred.JobClient: Combine output records=218990 >> 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory (bytes) >> snapshot=1179656192 >> 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output records=128513 >> 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory (bytes) >> snapshot=22992117760 >> 13/07/11 15:31:32 INFO mapred.JobClient: Map output records=1174991 >> >> from web interface (http://n1:50030/) I saw that both (n1 and n2 ) were >> used without any errors. >> >> Problems appear if I try to use following commands in master (n1): >> >> [hduser@n1 ~]$hadoop jar >> mahout-distribution-0.7/**mahout-examples-0.7-job.jar >> org.apache.mahout.classifier.**df.mapreduce.BuildForest >> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds >> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1 >> >> SLF4J: Class path contains multiple SLF4J bindings. >> SLF4J: Found binding in [file:/usr/local/hadoop-1.0.4/**org/slf4j/impl/** >> StaticLoggerBinder.class] >> SLF4J: Found binding in [jar:file:/usr/local/hadoop-1.** >> 0.4/lib/slf4j-log4j12-1.4.3.**jar!/org/slf4j/impl/** >> StaticLoggerBinder.class] >> SLF4J: See >> http://www.slf4j.org/codes.**html#multiple_bindings<http://www.slf4j.org/codes.html#multiple_bindings>for >> an explanation. >> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred >> implementation >> 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the forest... >> 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set. User >> classes may not be found. See JobConf(Class) or JobConf#setJar(String). >> 13/07/11 15:36:50 INFO input.FileInputFormat: Total input paths to >> process : 1 >> 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the native-hadoop >> library >> 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native library not loaded >> 13/07/11 15:36:50 INFO mapred.JobClient: Running job: >> job_201307111355_0016 >> 13/07/11 15:36:51 INFO mapred.JobClient: map 0% reduce 0% >> 13/07/11 15:37:09 INFO mapred.JobClient: map 7% reduce 0% >> 13/07/11 15:37:11 INFO mapred.JobClient: Task Id : >> attempt_201307111355_0016_m_**000002_0, Status : FAILED >> java.lang.RuntimeException: java.lang.**ClassNotFoundException: >> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper >> at org.apache.hadoop.conf.**Configuration.getClass(** >> Configuration.java:867) >> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(** >> JobContext.java:199) >> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.** >> java:719) >> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370) >> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255) >> at java.security.**AccessController.doPrivileged(**Native Method) >> at javax.security.auth.Subject.**doAs(Subject.java:415) >> at org.apache.hadoop.security.**UserGroupInformation.doAs(** >> UserGroupInformation.java:**1121) >> at org.apache.hadoop.mapred.**Child.main(Child.java:249) >> Caused by: java.lang.**ClassNotFoundException: >> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper >> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366) >> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355) >> at java.security.**AccessController.doPrivileged(**Native Method) >> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354) >> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424) >> at sun.misc.Launcher$**AppClassLoader.loadClass(** >> Launcher.java:308) >> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357) >> at java.lang.Class.forName0(**Native Method) >> at java.lang.Class.forName(Class.**java:270) >> at org.apache.hadoop.conf.**Configuration.getClassByName(** >> Configuration.java:820) >> at org.apache.hadoop.conf.**Configuration.getClass(** >> Configuration.java:865) >> ... 8 more >> >> ... >> ... >> ... >> >> 13/07/11 15:37:37 INFO mapred.JobClient: map 61% reduce 0% >> 13/07/11 15:37:46 INFO mapred.JobClient: map 76% reduce 0% >> 13/07/11 15:37:55 INFO mapred.JobClient: map 92% reduce 0% >> 13/07/11 15:38:01 INFO mapred.JobClient: map 100% reduce 0% >> 13/07/11 15:38:06 INFO mapred.JobClient: Job complete: >> job_201307111355_0016 >> 13/07/11 15:38:06 INFO mapred.JobClient: Counters: 20 >> 13/07/11 15:38:06 INFO mapred.JobClient: Job Counters >> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=184724 >> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all >> reduces waiting after reserving slots (ms)=0 >> 13/07/11 15:38:06 INFO mapred.JobClient: Total time spent by all maps >> waiting after reserving slots (ms)=0 >> 13/07/11 15:38:06 INFO mapred.JobClient: Rack-local map tasks=13 >> 13/07/11 15:38:06 INFO mapred.JobClient: Launched map tasks=22 >> 13/07/11 15:38:06 INFO mapred.JobClient: Data-local map tasks=9 >> 13/07/11 15:38:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 >> 13/07/11 15:38:06 INFO mapred.JobClient: File Output Format Counters >> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Written=184520 >> 13/07/11 15:38:06 INFO mapred.JobClient: FileSystemCounters >> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_READ=23444612 >> 13/07/11 15:38:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=294037 >> 13/07/11 15:38:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184520 >> 13/07/11 15:38:06 INFO mapred.JobClient: File Input Format Counters >> 13/07/11 15:38:06 INFO mapred.JobClient: Bytes Read=22952913 >> 13/07/11 15:38:06 INFO mapred.JobClient: Map-Reduce Framework >> 13/07/11 15:38:06 INFO mapred.JobClient: Map input records=204091 >> 13/07/11 15:38:06 INFO mapred.JobClient: Physical memory (bytes) >> snapshot=1402363904 >> 13/07/11 15:38:06 INFO mapred.JobClient: Spilled Records=0 >> 13/07/11 15:38:06 INFO mapred.JobClient: CPU time spent (ms)=26640 >> 13/07/11 15:38:06 INFO mapred.JobClient: Total committed heap usage >> (bytes)=1333985280 >> 13/07/11 15:38:06 INFO mapred.JobClient: Virtual memory (bytes) >> snapshot=37349126144 >> 13/07/11 15:38:06 INFO mapred.JobClient: Map output records=1 >> 13/07/11 15:38:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573 >> 13/07/11 15:38:07 INFO common.HadoopUtil: Deleting >> hdfs://master:54310/user/**hduser/bal_ee_2009_out >> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Build Time: 0h 1m 16s 977 >> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest num Nodes: 10700 >> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean num Nodes: 10700 >> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Forest mean max Depth: 17 >> 13/07/11 15:38:07 INFO mapreduce.BuildForest: Storing the forest in: >> bal_ee_2009_out/forest.seq >> >> From web console I can see that n1 tasks are fine but n2 tasks failed: >> >> java.lang.RuntimeException: java.lang.**ClassNotFoundException: >> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper >> at org.apache.hadoop.conf.**Configuration.getClass(** >> Configuration.java:867) >> at org.apache.hadoop.mapreduce.**JobContext.getMapperClass(** >> JobContext.java:199) >> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.** >> java:719) >> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370) >> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255) >> at java.security.**AccessController.doPrivileged(**Native Method) >> at javax.security.auth.Subject.**doAs(Subject.java:415) >> at org.apache.hadoop.security.**UserGroupInformation.doAs(** >> UserGroupInformation.java:**1121) >> at org.apache.hadoop.mapred.**Child.main(Child.java:249) >> Caused by: java.lang.**ClassNotFoundException: >> org.apache.mahout.classifier.**df.mapreduce.partial.**Step1Mapper >> at java.net.URLClassLoader$1.run(**URLClassLoader.java:366) >> at java.net.URLClassLoader$1.run(**URLClassLoader.java:355) >> at java.security.**AccessController.doPrivileged(**Native Method) >> at java.net.URLClassLoader.**findClass(URLClassLoader.java:**354) >> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**424) >> at sun.misc.Launcher$**AppClassLoader.loadClass(** >> Launcher.java:308) >> at java.lang.ClassLoader.**loadClass(ClassLoader.java:**357) >> at java.lang.Class.forName0(**Native Method) >> at java.lang.Class.forName(Class.**java:270) >> at org.apache.hadoop.conf.**Configuration.getClassByName(** >> Configuration.java:820) >> at org.apache.hadoop.conf.**Configuration.getClass(** >> Configuration.java:865) >> ... 8 more >> >> >> But If I run same command from n2 >> [hduser@n2 ~]$ hadoop jar >> mahout-distribution-0.7/**mahout-examples-0.7-job.jar >> org.apache.mahout.classifier.**df.mapreduce.BuildForest >> -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds >> testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1 >> >> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Partial Mapred >> implementation >> 13/07/11 15:41:20 INFO mapreduce.BuildForest: Building the forest... >> 13/07/11 15:41:21 INFO input.FileInputFormat: Total input paths to >> process : 1 >> 13/07/11 15:41:21 INFO util.NativeCodeLoader: Loaded the native-hadoop >> library >> 13/07/11 15:41:21 WARN snappy.LoadSnappy: Snappy native library not loaded >> 13/07/11 15:41:21 INFO mapred.JobClient: Running job: >> job_201307111355_0017 >> 13/07/11 15:41:22 INFO mapred.JobClient: map 0% reduce 0% >> 13/07/11 15:41:43 INFO mapred.JobClient: map 7% reduce 0% >> 13/07/11 15:41:46 INFO mapred.JobClient: map 15% reduce 0% >> 13/07/11 15:41:49 INFO mapred.JobClient: map 38% reduce 0% >> 13/07/11 15:41:52 INFO mapred.JobClient: map 46% reduce 0% >> 13/07/11 15:42:01 INFO mapred.JobClient: map 61% reduce 0% >> 13/07/11 15:42:02 INFO mapred.JobClient: map 76% reduce 0% >> 13/07/11 15:42:09 INFO mapred.JobClient: map 84% reduce 0% >> 13/07/11 15:42:13 INFO mapred.JobClient: map 100% reduce 0% >> 13/07/11 15:42:19 INFO mapred.JobClient: Job complete: >> job_201307111355_0017 >> 13/07/11 15:42:19 INFO mapred.JobClient: Counters: 20 >> 13/07/11 15:42:19 INFO mapred.JobClient: Job Counters >> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=138911 >> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all >> reduces waiting after reserving slots (ms)=0 >> 13/07/11 15:42:19 INFO mapred.JobClient: Total time spent by all maps >> waiting after reserving slots (ms)=0 >> 13/07/11 15:42:19 INFO mapred.JobClient: Rack-local map tasks=6 >> 13/07/11 15:42:19 INFO mapred.JobClient: Launched map tasks=13 >> 13/07/11 15:42:19 INFO mapred.JobClient: Data-local map tasks=7 >> 13/07/11 15:42:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 >> 13/07/11 15:42:19 INFO mapred.JobClient: File Output Format Counters >> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Written=184956 >> 13/07/11 15:42:19 INFO mapred.JobClient: FileSystemCounters >> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_READ=23444612 >> 13/07/11 15:42:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=296026 >> 13/07/11 15:42:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=184956 >> 13/07/11 15:42:19 INFO mapred.JobClient: File Input Format Counters >> 13/07/11 15:42:19 INFO mapred.JobClient: Bytes Read=22952913 >> 13/07/11 15:42:19 INFO mapred.JobClient: Map-Reduce Framework >> 13/07/11 15:42:19 INFO mapred.JobClient: Map input records=204091 >> 13/07/11 15:42:19 INFO mapred.JobClient: Physical memory (bytes) >> snapshot=1370255360 >> 13/07/11 15:42:19 INFO mapred.JobClient: Spilled Records=0 >> 13/07/11 15:42:19 INFO mapred.JobClient: CPU time spent (ms)=27990 >> 13/07/11 15:42:19 INFO mapred.JobClient: Total committed heap usage >> (bytes)=1292500992 >> 13/07/11 15:42:19 INFO mapred.JobClient: Virtual memory (bytes) >> snapshot=37360082944 >> 13/07/11 15:42:19 INFO mapred.JobClient: Map output records=1 >> 13/07/11 15:42:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=1573 >> 13/07/11 15:42:20 INFO common.HadoopUtil: Deleting >> hdfs://master:54310/user/**hduser/bal_ee_2009_out >> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Build Time: 0h 0m 59s 386 >> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest num Nodes: 10698 >> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean num Nodes: 10698 >> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Forest mean max Depth: 16 >> 13/07/11 15:42:20 INFO mapreduce.BuildForest: Storing the forest in: >> bal_ee_2009_out/forest.seq >> >> Both (n1 and n2) are used and from web console I can see that there are >> no errors. >> >> Is there any explanations why I am getting errors when I run command from >> master? >> >> >> -- >> Regards, Margus (Margusja) Roo >> +372 51 48 780 >> http://margus.roo.ee >> skype: margusja >> -----BEGIN PUBLIC KEY----- >> MIGfMA0GCSqGSIb3DQEBAQUAA4GNAD**CBiQKBgQCvbeg7LwEC2SCpAEewwpC3**ajxE >> 5ZsRMCB77L8bae9G7TslgLkoIzo9yO**jPdx2NN6DllKbV65UjTay43uUDyql9**g3tl >> RhiJIcoAExkSTykWqAIPR88LfilLy1**JlQ+0RD8OXiWOVVQfhOHpQ0R/**jcAkM2lZa >> BjM8j36yJvoBVsfOHQIDAQAB >> -----END PUBLIC KEY----- >> >>
