Re: Task failure in slave node
Hi, It seems mahout-examples-0.7-job.jar is depending on other jars/classes. While running Job Tasks it is not able to find those classes in the classpath and failing those tasks. You need to provide the dependent jar files while submitting/running Job. Thanks Devaraj k -- View this message in context: http://lucene.472066.n3.nabble.com/Task-failure-in-slave-node-tp4077284p4077290.html Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Task failure in slave node
Hi I have tow nodes: n1 (master, salve) and n2 (slave) after set up I ran wordcount example and it worked fine: [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-examples-1.0.4.jar wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output 13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to process : 7 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded 13/07/11 15:30:44 INFO mapred.JobClient: Running job: job_201307111355_0015 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0% 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0% 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0% 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0% 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0% 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23% 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100% 13/07/11 15:31:32 INFO mapred.JobClient: Job complete: job_201307111355_0015 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce tasks=1 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map tasks=3 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map tasks=4 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format Counters 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format Counters 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework 13/07/11 15:31:32 INFO mapred.JobClient: Map output materialized bytes=3157469 13/07/11 15:31:32 INFO mapred.JobClient: Map input records=137146 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle bytes=2904836 13/07/11 15:31:32 INFO mapred.JobClient: Spilled Records=594764 13/07/11 15:31:32 INFO mapred.JobClient: Map output bytes=11435849 13/07/11 15:31:32 INFO mapred.JobClient: Total committed heap usage (bytes)=1128136704 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent (ms)=18230 13/07/11 15:31:32 INFO mapred.JobClient: Combine input records=1174991 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input records=218990 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input groups=128513 13/07/11 15:31:32 INFO mapred.JobClient: Combine output records=218990 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory (bytes) snapshot=1179656192 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output records=128513 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory (bytes) snapshot=22992117760 13/07/11 15:31:32 INFO mapred.JobClient: Map output records=1174991 from web interface (http://n1:50030/) I saw that both (n1 and n2 ) were used without any errors. Problems appear if I try to use following commands in master (n1): [hduser@n1 ~]$hadoop jar mahout-distribution-0.7/mahout-examples-0.7-job.jar org.apache.mahout.classifier.df.mapreduce.BuildForest -Dmapred.max.split.size=1874231 -p -d testdata/bal_ee_2009.csv -ds testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [file:/usr/local/hadoop-1.0.4/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop-1.0.4/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred implementation 13/07/11 15:36:50 INFO mapreduce.BuildForest: Building the forest... 13/07/11 15:36:50 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). 13/07/11 15:36:50 INFO input.FileInputFormat: Total input paths to process : 1 13/07/11 15:36:50 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/07/11 15:36:50 WARN snappy.LoadSnappy: Snappy native
Re: Task failure in slave node
sorry for typo, mahout, not mahou. sent from mobile On Jul 11, 2013 9:40 PM, Azuryy Yu azury...@gmail.com wrote: hi, put all mahou jars under hadoop_home/lib, then restart cluster. On Jul 11, 2013 8:45 PM, Margusja mar...@roo.ee wrote: Hi I have tow nodes: n1 (master, salve) and n2 (slave) after set up I ran wordcount example and it worked fine: [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-**examples-1.0.4.jar wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output 13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to process : 7 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded 13/07/11 15:30:44 INFO mapred.JobClient: Running job: job_201307111355_0015 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0% 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0% 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0% 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0% 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0% 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23% 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100% 13/07/11 15:31:32 INFO mapred.JobClient: Job complete: job_201307111355_0015 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce tasks=1 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map tasks=3 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map tasks=4 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format Counters 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format Counters 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework 13/07/11 15:31:32 INFO mapred.JobClient: Map output materialized bytes=3157469 13/07/11 15:31:32 INFO mapred.JobClient: Map input records=137146 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle bytes=2904836 13/07/11 15:31:32 INFO mapred.JobClient: Spilled Records=594764 13/07/11 15:31:32 INFO mapred.JobClient: Map output bytes=11435849 13/07/11 15:31:32 INFO mapred.JobClient: Total committed heap usage (bytes)=1128136704 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent (ms)=18230 13/07/11 15:31:32 INFO mapred.JobClient: Combine input records=1174991 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input records=218990 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input groups=128513 13/07/11 15:31:32 INFO mapred.JobClient: Combine output records=218990 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory (bytes) snapshot=1179656192 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output records=128513 13/07/11 15:31:32 INFO mapred.JobClient: Virtual memory (bytes) snapshot=22992117760 13/07/11 15:31:32 INFO mapred.JobClient: Map output records=1174991 from web interface (http://n1:50030/) I saw that both (n1 and n2 ) were used without any errors. Problems appear if I try to use following commands in master (n1): [hduser@n1 ~]$hadoop jar mahout-distribution-0.7/**mahout-examples-0.7-job.jar org.apache.mahout.classifier.**df.mapreduce.BuildForest -Dmapred.max.split.size=**1874231 -p -d testdata/bal_ee_2009.csv -ds testdata/bal_ee_2009.csv.info -sl 10 -o bal_ee_2009_out -t 1 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [file:/usr/local/hadoop-1.0.4/**org/slf4j/impl/** StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop-1.** 0.4/lib/slf4j-log4j12-1.4.3.**jar!/org/slf4j/impl/** StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.**html#multiple_bindingshttp://www.slf4j.org/codes.html#multiple_bindingsfor an explanation. 13/07/11 15:36:50 INFO mapreduce.BuildForest: Partial Mapred implementation 13/07/11 15:36:50 INFO mapreduce.BuildForest:
Re: Task failure in slave node
Than you, it resolved the problem. Funny, I don't remember that I copied mahout libs to n1 hadoop but there they are. Tervitades, Margus (Margusja) Roo +372 51 48 780 http://margus.roo.ee skype: margusja -BEGIN PUBLIC KEY- MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE 5ZsRMCB77L8bae9G7TslgLkoIzo9yOjPdx2NN6DllKbV65UjTay43uUDyql9g3tl RhiJIcoAExkSTykWqAIPR88LfilLy1JlQ+0RD8OXiWOVVQfhOHpQ0R/jcAkM2lZa BjM8j36yJvoBVsfOHQIDAQAB -END PUBLIC KEY- On 7/11/13 4:41 PM, Azuryy Yu wrote: sorry for typo, mahout, not mahou. sent from mobile On Jul 11, 2013 9:40 PM, Azuryy Yu azury...@gmail.com mailto:azury...@gmail.com wrote: hi, put all mahou jars under hadoop_home/lib, then restart cluster. On Jul 11, 2013 8:45 PM, Margusja mar...@roo.ee mailto:mar...@roo.ee wrote: Hi I have tow nodes: n1 (master, salve) and n2 (slave) after set up I ran wordcount example and it worked fine: [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-examples-1.0.4.jar wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output 13/07/11 15:30:44 INFO input.FileInputFormat: Total input paths to process : 7 13/07/11 15:30:44 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/07/11 15:30:44 WARN snappy.LoadSnappy: Snappy native library not loaded 13/07/11 15:30:44 INFO mapred.JobClient: Running job: job_201307111355_0015 13/07/11 15:30:45 INFO mapred.JobClient: map 0% reduce 0% 13/07/11 15:31:03 INFO mapred.JobClient: map 42% reduce 0% 13/07/11 15:31:06 INFO mapred.JobClient: map 57% reduce 0% 13/07/11 15:31:09 INFO mapred.JobClient: map 71% reduce 0% 13/07/11 15:31:15 INFO mapred.JobClient: map 100% reduce 0% 13/07/11 15:31:18 INFO mapred.JobClient: map 100% reduce 23% 13/07/11 15:31:27 INFO mapred.JobClient: map 100% reduce 100% 13/07/11 15:31:32 INFO mapred.JobClient: Job complete: job_201307111355_0015 13/07/11 15:31:32 INFO mapred.JobClient: Counters: 30 13/07/11 15:31:32 INFO mapred.JobClient: Job Counters 13/07/11 15:31:32 INFO mapred.JobClient: Launched reduce tasks=1 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67576 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/07/11 15:31:32 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/07/11 15:31:32 INFO mapred.JobClient: Rack-local map tasks=3 13/07/11 15:31:32 INFO mapred.JobClient: Launched map tasks=7 13/07/11 15:31:32 INFO mapred.JobClient: Data-local map tasks=4 13/07/11 15:31:32 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=21992 13/07/11 15:31:32 INFO mapred.JobClient: File Output Format Counters 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Written=1412505 13/07/11 15:31:32 INFO mapred.JobClient: FileSystemCounters 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_READ=5414195 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_READ=6950820 13/07/11 15:31:32 INFO mapred.JobClient: FILE_BYTES_WRITTEN=8744993 13/07/11 15:31:32 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1412505 13/07/11 15:31:32 INFO mapred.JobClient: File Input Format Counters 13/07/11 15:31:32 INFO mapred.JobClient: Bytes Read=6950001 13/07/11 15:31:32 INFO mapred.JobClient: Map-Reduce Framework 13/07/11 15:31:32 INFO mapred.JobClient: Map output materialized bytes=3157469 13/07/11 15:31:32 INFO mapred.JobClient: Map input records=137146 13/07/11 15:31:32 INFO mapred.JobClient: Reduce shuffle bytes=2904836 13/07/11 15:31:32 INFO mapred.JobClient: Spilled Records=594764 13/07/11 15:31:32 INFO mapred.JobClient: Map output bytes=11435849 13/07/11 15:31:32 INFO mapred.JobClient: Total committed heap usage (bytes)=1128136704 13/07/11 15:31:32 INFO mapred.JobClient: CPU time spent (ms)=18230 13/07/11 15:31:32 INFO mapred.JobClient: Combine input records=1174991 13/07/11 15:31:32 INFO mapred.JobClient: SPLIT_RAW_BYTES=819 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input records=218990 13/07/11 15:31:32 INFO mapred.JobClient: Reduce input groups=128513 13/07/11 15:31:32 INFO mapred.JobClient: Combine output records=218990 13/07/11 15:31:32 INFO mapred.JobClient: Physical memory (bytes) snapshot=1179656192 13/07/11 15:31:32 INFO mapred.JobClient: Reduce output records=128513 13/07/11
