Hi, Thanks a lot for your response.
Unfortunately, I run into the same problem though. What do you mean by "python binary"? This is what I have in the very first line of both scripts: #!/usr/bin/python Any ideas? On Fri, May 24, 2013 at 7:41 PM, Jitendra Yadav <[email protected]>wrote: > Hi, > > I have run Michael's python map reduce example several times without any > issue. > > I think this issue is related to your file path 'mapper.py'. you are > using python binary? > > try this, > > hadoop jar > /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/hadoop-streaming-1.1.2.jar > \ > -input /user/yyy/20417-8.txt \ > -output /user/yyy/output \ > -file /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/mapper.py \ > -mapper /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/mapper.py \ > -file /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/reducer.py \ > -reducer /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/reducer.py > > > Thanks~ > > On Fri, May 24, 2013 at 10:12 PM, Adamantios Corais < > [email protected]> wrote: > >> I tried this nice example: >> http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/ >> >> The python scripts work pretty fine from my laptop (through terminal), >> but they don't when I execute them on the CDH3 (Pseudo-Distributed Mode). >> >> Any ideas? >> >> hadoop jar >> /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/hadoop-streaming-1.1.2.jar >> \ >> -input /user/yyy/20417-8.txt \ >> -output /user/yyy/output \ >> -file /usr/bin/python >> /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/mapper.py \ >> -mapper /usr/bin/python >> /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/mapper.py \ >> -file /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/reducer.py \ >> -reducer /home/yyy/Dropbox/Private/xxx/Projects/task_week_22/reducer.py >> >> ----------------------- >> >> 2013-05-24 18:21:12,232 INFO org.apache.hadoop.util.NativeCodeLoader: >> Loaded the native-hadoop library >> 2013-05-24 18:21:12,569 INFO >> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating >> symlink: >> /var/lib/hadoop-0.20/cache/mapred/mapred/local/taskTracker/yyy/jobcache/job_201305160832_0020/jars/job.jar >> <- >> /var/lib/hadoop-0.20/cache/mapred/mapred/local/taskTracker/yyy/jobcache/job_201305160832_0020/attempt_201305160832_0020_m_000000_0/work/job.jar >> 2013-05-24 18:21:12,586 INFO >> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating >> symlink: >> /var/lib/hadoop-0.20/cache/mapred/mapred/local/taskTracker/yyy/jobcache/job_201305160832_0020/jars/.job.jar.crc >> <- >> /var/lib/hadoop-0.20/cache/mapred/mapred/local/taskTracker/yyy/jobcache/job_201305160832_0020/attempt_201305160832_0020_m_000000_0/work/.job.jar.crc >> 2013-05-24 18:21:12,717 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: >> Initializing JVM Metrics with processName=MAP, sessionId= >> 2013-05-24 18:21:13,062 INFO org.apache.hadoop.util.ProcessTree: setsid >> exited with exit code 0 >> 2013-05-24 18:21:13,087 INFO org.apache.hadoop.mapred.Task: Using >> ResourceCalculatorPlugin : >> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1358f03 >> 2013-05-24 18:21:13,452 WARN >> org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library is >> available >> 2013-05-24 18:21:13,452 INFO >> org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library >> loaded >> 2013-05-24 18:21:13,464 INFO org.apache.hadoop.mapred.MapTask: >> numReduceTasks: 1 >> 2013-05-24 18:21:13,477 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb >> = 100 >> 2013-05-24 18:21:13,635 INFO org.apache.hadoop.mapred.MapTask: data >> buffer = 79691776/99614720 >> 2013-05-24 18:21:13,635 INFO org.apache.hadoop.mapred.MapTask: record >> buffer = 262144/327680 >> 2013-05-24 18:21:13,724 INFO org.mortbay.log: Logging to >> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via >> org.mortbay.log.Slf4jLog >> 2013-05-24 18:21:13,733 INFO org.apache.hadoop.streaming.PipeMapRed: >> PipeMapRed exec [mapper.py] >> 2013-05-24 18:21:13,783 ERROR org.apache.hadoop.streaming.PipeMapRed: >> configuration exception >> java.io.IOException: Cannot run program "mapper.py": java.io.IOException: >> error=2, No such file or directory >> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) >> at >> org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214) >> at >> org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) >> at >> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) >> at >> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) >> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) >> at >> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) >> at >> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) >> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) >> at org.apache.hadoop.mapred.Child$4.run(Child.java:266) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:396) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278) >> at org.apache.hadoop.mapred.Child.main(Child.java:260) >> Caused by: java.io.IOException: java.io.IOException: error=2, No such >> file or directory >> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) >> at java.lang.ProcessImpl.start(ProcessImpl.java:65) >> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) >> ... 24 more >> 2013-05-24 18:21:13,869 INFO org.apache.hadoop.mapred.TaskLogsTruncater: >> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 >> 2013-05-24 18:21:13,871 WARN org.apache.hadoop.mapred.Child: Error >> running child >> java.lang.RuntimeException: Error in configuring object >> at >> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) >> at >> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) >> at >> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) >> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) >> at org.apache.hadoop.mapred.Child$4.run(Child.java:266) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:396) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278) >> at org.apache.hadoop.mapred.Child.main(Child.java:260) >> Caused by: java.lang.reflect.InvocationTargetException >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) >> ... 9 more >> Caused by: java.lang.RuntimeException: Error in configuring object >> at >> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) >> at >> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) >> at >> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) >> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) >> ... 14 more >> Caused by: java.lang.reflect.InvocationTargetException >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) >> ... 17 more >> Caused by: java.lang.RuntimeException: configuration exception >> at >> org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:230) >> at >> org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) >> ... 22 more >> Caused by: java.io.IOException: Cannot run program "mapper.py": >> java.io.IOException: error=2, No such file or directory >> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) >> at >> org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214) >> ... 23 more >> Caused by: java.io.IOException: java.io.IOException: error=2, No such >> file or directory >> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) >> at java.lang.ProcessImpl.start(ProcessImpl.java:65) >> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) >> ... 24 more >> 2013-05-24 18:21:13,879 INFO org.apache.hadoop.mapred.Task: Runnning >> cleanup for the task >> > >
