Re: pipes on hadoop 2.2.0 crashes
I tested that example as well and I'm getting the same exception with this in stderr: Hadoop Pipes Exception: failed to open hdfs://compute-0-7-2:54310/in/file at /home/scaino/hadoop-2.2.0-maven/hadoop-tools/hadoop-pipes/src/main/native/examples/impl/wordcount-nopipe.cc:82 in WordCountReader I found out that wordcount-nopipe is picky for its input format, here is the full explanation: http://lucene.472066.n3.nabble.com/pipe-application-error-td650185.html On 17 December 2013 17:36, Mauro Del Rio mdrio1...@gmail.com wrote: Ok, I had some problems with configuration and host resolution and I fixed them. I was able to run successfully the simple wordcount example, but I failed running wordcount-nopipe.cc. This is the stack trace: Error: java.io.IOException: pipe child exception at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:226) at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:102) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157) Caused by: java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:250) at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308) at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329) at org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:125) Any suggestions? 2013/12/17 Silvina Caíno Lores silvi.ca...@gmail.com I'm having similar problems with pipes, mostly because of issues with the native shared libraries that leave the job stuck either at 0%-0% or before launch (because the resource manager gets stuck as well and crashes). I found that out by looking at the stderr logs by the way. Let us know if you progress, good luck! Best, Silvina On 16 December 2013 19:32, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: You should navigate to the ResourceManager UI following the link and see what is happening on the ResourceManager as well as the application-master. Check if any nodes are active first. Then look at ResourceManager and NodeManager logs. +Vinod On Dec 16, 2013, at 10:29 AM, Mauro Del Rio mdrio1...@gmail.com wrote: I installed hadoop 2.2.0 on a small cluster, just two nodes. I run a simple wordcount in c++ with pipes, this time there was no exception, but the job didn't finish. This is the output on the shell where I launched pipes: mauro@mauro-VirtualBox:~/hadoop-2.2.0$ bin/mapred pipes -program wc -input test.sh -output out 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at / 0.0.0.0:9052 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at / 0.0.0.0:9052 13/12/16 18:51:41 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 13/12/16 18:51:41 INFO mapred.FileInputFormat: Total input paths to process : 1 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: number of splits:2 13/12/16 18:51:41 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.pipes.user.inputformat is deprecated. Instead, use mapreduce.pipes.inputformat 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 13/12/16 18:51:41 INFO Configuration.deprecation: hadoop.pipes.partitioner is deprecated. Instead, use mapreduce.pipes.partitioner 13/12/16 18:51:41 INFO Configuration.deprecation: hadoop.pipes.executable is deprecated. Instead, use mapreduce.pipes.executable 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps
Re: pipes on hadoop 2.2.0 crashes
I'm having similar problems with pipes, mostly because of issues with the native shared libraries that leave the job stuck either at 0%-0% or before launch (because the resource manager gets stuck as well and crashes). I found that out by looking at the stderr logs by the way. Let us know if you progress, good luck! Best, Silvina On 16 December 2013 19:32, Vinod Kumar Vavilapalli vino...@hortonworks.comwrote: You should navigate to the ResourceManager UI following the link and see what is happening on the ResourceManager as well as the application-master. Check if any nodes are active first. Then look at ResourceManager and NodeManager logs. +Vinod On Dec 16, 2013, at 10:29 AM, Mauro Del Rio mdrio1...@gmail.com wrote: I installed hadoop 2.2.0 on a small cluster, just two nodes. I run a simple wordcount in c++ with pipes, this time there was no exception, but the job didn't finish. This is the output on the shell where I launched pipes: mauro@mauro-VirtualBox:~/hadoop-2.2.0$ bin/mapred pipes -program wc -input test.sh -output out 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at / 0.0.0.0:9052 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at / 0.0.0.0:9052 13/12/16 18:51:41 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 13/12/16 18:51:41 INFO mapred.FileInputFormat: Total input paths to process : 1 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: number of splits:2 13/12/16 18:51:41 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.pipes.user.inputformat is deprecated. Instead, use mapreduce.pipes.inputformat 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 13/12/16 18:51:41 INFO Configuration.deprecation: hadoop.pipes.partitioner is deprecated. Instead, use mapreduce.pipes.partitioner 13/12/16 18:51:41 INFO Configuration.deprecation: hadoop.pipes.executable is deprecated. Instead, use mapreduce.pipes.executable 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1387213974967_0003 13/12/16 18:51:42 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources. 13/12/16 18:51:42 INFO impl.YarnClientImpl: Submitted application application_1387213974967_0003 to ResourceManager at /0.0.0.0:9052 13/12/16 18:51:42 INFO mapreduce.Job: The url to track the job: http://mauro-VirtualBox:8088/proxy/application_1387213974967_0003/http://mauro-virtualbox:8088/proxy/application_1387213974967_0003/ 13/12/16 18:51:42 INFO mapreduce.Job: Running job: job_1387213974967_0003 The job status from bin/mapred job -list is PREP. I didn't find any interesting information in logs file. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: pipes on hadoop 2.2.0 crashes
Ok, I had some problems with configuration and host resolution and I fixed them. I was able to run successfully the simple wordcount example, but I failed running wordcount-nopipe.cc. This is the stack trace: Error: java.io.IOException: pipe child exception at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:226) at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:102) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157) Caused by: java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:250) at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308) at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329) at org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:125) Any suggestions? 2013/12/17 Silvina Caíno Lores silvi.ca...@gmail.com I'm having similar problems with pipes, mostly because of issues with the native shared libraries that leave the job stuck either at 0%-0% or before launch (because the resource manager gets stuck as well and crashes). I found that out by looking at the stderr logs by the way. Let us know if you progress, good luck! Best, Silvina On 16 December 2013 19:32, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: You should navigate to the ResourceManager UI following the link and see what is happening on the ResourceManager as well as the application-master. Check if any nodes are active first. Then look at ResourceManager and NodeManager logs. +Vinod On Dec 16, 2013, at 10:29 AM, Mauro Del Rio mdrio1...@gmail.com wrote: I installed hadoop 2.2.0 on a small cluster, just two nodes. I run a simple wordcount in c++ with pipes, this time there was no exception, but the job didn't finish. This is the output on the shell where I launched pipes: mauro@mauro-VirtualBox:~/hadoop-2.2.0$ bin/mapred pipes -program wc -input test.sh -output out 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at / 0.0.0.0:9052 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at / 0.0.0.0:9052 13/12/16 18:51:41 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 13/12/16 18:51:41 INFO mapred.FileInputFormat: Total input paths to process : 1 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: number of splits:2 13/12/16 18:51:41 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.pipes.user.inputformat is deprecated. Instead, use mapreduce.pipes.inputformat 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 13/12/16 18:51:41 INFO Configuration.deprecation: hadoop.pipes.partitioner is deprecated. Instead, use mapreduce.pipes.partitioner 13/12/16 18:51:41 INFO Configuration.deprecation: hadoop.pipes.executable is deprecated. Instead, use mapreduce.pipes.executable 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1387213974967_0003 13/12/16 18:51:42 INFO mapred.YARNRunner: Job jar is not present.
Re: pipes on hadoop 2.2.0 crashes
Ok, I will try. Il 13/dic/2013 19:23 Vinod Kumar Vavilapalli vino...@hortonworks.com ha scritto: Could it just be LocalJobRunner? Can you try it on a cluster? We've tested pipes on clusters, so will be surprised if it doesn't work there. Thanks, +Vinod On Dec 13, 2013, at 7:44 AM, Mauro Del Rio mdrio1...@gmail.com wrote: Hi, I tried to run a simple test with pipes, but it crashes. java.lang.Exception: java.lang.NullPointerException at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:403) Caused by: java.lang.NullPointerException at org.apache.hadoop.mapred.pipes.Application.init(Application.java:104) at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:69) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 13/12/13 16:38:03 INFO mapreduce.Job: Job job_local1570213319_0001 running in uber mode : false 13/12/13 16:38:03 INFO mapreduce.Job: map 0% reduce 0% 13/12/13 16:38:03 INFO mapreduce.Job: Job job_local1570213319_0001 failed with state FAILED due to: NA 13/12/13 16:38:03 INFO mapreduce.Job: Counters: 0 Exception in thread main java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:264) at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:503) at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:518) The line Application.java:104 is: byte[] password = jobToken.getPassword(); so jobToken seems to be null. It does not depend upon c++ code, since the error occurs before it is launched. I run it on Ubuntu 12.04 32 bit using the binary tarball. Any idea why it doesn't work? -- Mauro CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: pipes on hadoop 2.2.0 crashes
I installed hadoop 2.2.0 on a small cluster, just two nodes. I run a simple wordcount in c++ with pipes, this time there was no exception, but the job didn't finish. This is the output on the shell where I launched pipes: mauro@mauro-VirtualBox:~/hadoop-2.2.0$ bin/mapred pipes -program wc -input test.sh -output out 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at / 0.0.0.0:9052 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at / 0.0.0.0:9052 13/12/16 18:51:41 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 13/12/16 18:51:41 INFO mapred.FileInputFormat: Total input paths to process : 1 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: number of splits:2 13/12/16 18:51:41 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.pipes.user.inputformat is deprecated. Instead, use mapreduce.pipes.inputformat 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 13/12/16 18:51:41 INFO Configuration.deprecation: hadoop.pipes.partitioner is deprecated. Instead, use mapreduce.pipes.partitioner 13/12/16 18:51:41 INFO Configuration.deprecation: hadoop.pipes.executable is deprecated. Instead, use mapreduce.pipes.executable 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1387213974967_0003 13/12/16 18:51:42 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources. 13/12/16 18:51:42 INFO impl.YarnClientImpl: Submitted application application_1387213974967_0003 to ResourceManager at /0.0.0.0:9052 13/12/16 18:51:42 INFO mapreduce.Job: The url to track the job: http://mauro-VirtualBox:8088/proxy/application_1387213974967_0003/ 13/12/16 18:51:42 INFO mapreduce.Job: Running job: job_1387213974967_0003 The job status from bin/mapred job -list is PREP. I didn't find any interesting information in logs file.
Re: pipes on hadoop 2.2.0 crashes
You should navigate to the ResourceManager UI following the link and see what is happening on the ResourceManager as well as the application-master. Check if any nodes are active first. Then look at ResourceManager and NodeManager logs. +Vinod On Dec 16, 2013, at 10:29 AM, Mauro Del Rio mdrio1...@gmail.com wrote: I installed hadoop 2.2.0 on a small cluster, just two nodes. I run a simple wordcount in c++ with pipes, this time there was no exception, but the job didn't finish. This is the output on the shell where I launched pipes: mauro@mauro-VirtualBox:~/hadoop-2.2.0$ bin/mapred pipes -program wc -input test.sh -output out 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:9052 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:9052 13/12/16 18:51:41 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 13/12/16 18:51:41 INFO mapred.FileInputFormat: Total input paths to process : 1 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: number of splits:2 13/12/16 18:51:41 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.pipes.user.inputformat is deprecated. Instead, use mapreduce.pipes.inputformat 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 13/12/16 18:51:41 INFO Configuration.deprecation: hadoop.pipes.partitioner is deprecated. Instead, use mapreduce.pipes.partitioner 13/12/16 18:51:41 INFO Configuration.deprecation: hadoop.pipes.executable is deprecated. Instead, use mapreduce.pipes.executable 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1387213974967_0003 13/12/16 18:51:42 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources. 13/12/16 18:51:42 INFO impl.YarnClientImpl: Submitted application application_1387213974967_0003 to ResourceManager at /0.0.0.0:9052 13/12/16 18:51:42 INFO mapreduce.Job: The url to track the job: http://mauro-VirtualBox:8088/proxy/application_1387213974967_0003/ 13/12/16 18:51:42 INFO mapreduce.Job: Running job: job_1387213974967_0003 The job status from bin/mapred job -list is PREP. I didn't find any interesting information in logs file. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
pipes on hadoop 2.2.0 crashes
Hi, I tried to run a simple test with pipes, but it crashes. java.lang.Exception: java.lang.NullPointerException at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:403) Caused by: java.lang.NullPointerException at org.apache.hadoop.mapred.pipes.Application.init(Application.java:104) at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:69) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 13/12/13 16:38:03 INFO mapreduce.Job: Job job_local1570213319_0001 running in uber mode : false 13/12/13 16:38:03 INFO mapreduce.Job: map 0% reduce 0% 13/12/13 16:38:03 INFO mapreduce.Job: Job job_local1570213319_0001 failed with state FAILED due to: NA 13/12/13 16:38:03 INFO mapreduce.Job: Counters: 0 Exception in thread main java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:264) at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:503) at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:518) The line Application.java:104 is: byte[] password = jobToken.getPassword(); so jobToken seems to be null. It does not depend upon c++ code, since the error occurs before it is launched. I run it on Ubuntu 12.04 32 bit using the binary tarball. Any idea why it doesn't work? -- Mauro
Re: pipes on hadoop 2.2.0 crashes
Could it just be LocalJobRunner? Can you try it on a cluster? We've tested pipes on clusters, so will be surprised if it doesn't work there. Thanks, +Vinod On Dec 13, 2013, at 7:44 AM, Mauro Del Rio mdrio1...@gmail.com wrote: Hi, I tried to run a simple test with pipes, but it crashes. java.lang.Exception: java.lang.NullPointerException at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:403) Caused by: java.lang.NullPointerException at org.apache.hadoop.mapred.pipes.Application.init(Application.java:104) at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:69) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 13/12/13 16:38:03 INFO mapreduce.Job: Job job_local1570213319_0001 running in uber mode : false 13/12/13 16:38:03 INFO mapreduce.Job: map 0% reduce 0% 13/12/13 16:38:03 INFO mapreduce.Job: Job job_local1570213319_0001 failed with state FAILED due to: NA 13/12/13 16:38:03 INFO mapreduce.Job: Counters: 0 Exception in thread main java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:264) at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:503) at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:518) The line Application.java:104 is: byte[] password = jobToken.getPassword(); so jobToken seems to be null. It does not depend upon c++ code, since the error occurs before it is launched. I run it on Ubuntu 12.04 32 bit using the binary tarball. Any idea why it doesn't work? -- Mauro -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
