Hi,
I have a question about how to use php-cgi script and streaming, can
anybody help me check it?
But it is ok when I replace this php-cgi script with a perl script of
same function.
But I'm sure the environment has no problem.
Should I pay special attention on php-cgi?
*
hadoop command:*
hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-CDH3B4.jar
-input /user/hadoop/facebook/citylife/tw/input
-output /user/hadoop/facebook/citylife/tw/output
-mapper "./test.php"
-reducer NONE
-file test.php
*file 'test.php':*
#!/usr/bin/php-cgi -q
<?php
echo "ok";
?>
*output:*
packageJobJar: [test.php,
/data/hadoop-0.20/cache/hadoop/hadoop-unjar8132304638271650302/] []
/tmp/streamjob4835473350761937152.jar tmpDir=null
11/09/15 06:42:28 INFO mapred.FileInputFormat: Total input paths to process : 1
11/09/15 06:42:28 INFO streaming.StreamJob: getLocalDirs():
[/data/hadoop-0.20/cache/hadoop/mapred/local]
11/09/15 06:42:28 INFO streaming.StreamJob: Running job: job_201105231856_111690
11/09/15 06:42:28 INFO streaming.StreamJob: To kill this job, run:
11/09/15 06:42:28 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job
-Dmapred.job.tracker=namenode.xingcloud.com:19001 -kill job_201105231856_111690
11/09/15 06:42:28 INFO streaming.StreamJob: Tracking URL:
http://namenode.xingcloud.com:50030/jobdetails.jsp?jobid=job_201105231856_111690
11/09/15 06:42:29 INFO streaming.StreamJob: map 0% reduce 0%
11/09/15 06:42:48 INFO streaming.StreamJob: map 100% reduce 100%
11/09/15 06:42:48 INFO streaming.StreamJob: To kill this job, run:
11/09/15 06:42:48 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job
-Dmapred.job.tracker=namenode.xingcloud.com:19001 -kill job_201105231856_111690
11/09/15 06:42:48 INFO streaming.StreamJob: Tracking URL:
http://namenode.xingcloud.com:50030/jobdetails.jsp?jobid=job_201105231856_111690
11/09/15 06:42:48 ERROR streaming.StreamJob: Job not successful. Error: NA
11/09/15 06:42:48 INFO streaming.StreamJob: killJob...Streaming Command Failed!
*Error message:*
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed
with code 255
at
org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at
org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:572)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:390)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324)
at org.apache.hadoop.mapred.Child$4.run(Child.java:240)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:234)
Thanks,
Pei
