Issues with mv command when used after store when using -param_file/-param
options
----------------------------------------------------------------------------------
Key: PIG-974
URL: https://issues.apache.org/jira/browse/PIG-974
Project: Pig
Issue Type: Bug
Affects Versions: 0.6.0
Environment: Hadoop 18 and 20
Reporter: Viraj Bhat
Fix For: 0.6.0
Attachments: studenttab10k
I have a Pig script which moves the final output to another HDFS directory to
signal completion, so that another Pig script can start working on these
results.
{code}
studenttab = LOAD '/user/viraj/studenttab10k' AS (name:chararray,
age:int,gpa:float);
X = GROUP studenttab by age;
Y = FOREACH X GENERATE group, COUNT(studenttab);
store Y into '$finalop' using PigStorage();
mv '$finalop' '$finalmove';
{code}
where "finalop" and "finalmove" are parameters used storing intermediate and
final results.
I run this script as this:
{code}
$shell> java -cp pig20.jar:/path/tohadoop/site.xml
-Dmapred.job.queue.name=default org.apache.pig.Main -M -param
finalop=/user/viraj/finaloutput -param finalmove=/user/viraj/finalmove
testmove.pig
{code}
or using the param_file option
{code}
$shell>java -cp pig20.jar:/path/tohadoop/site.xml
-Dmapred.job.queue.name=default org.apache.pig.Main -M -param_file
moveparamfile testmove.pig
{code}
================================================================================
The underlying Map Reduce jobs run well but the move command seems to be
failing:
================================================================================
2009-09-23 23:26:21,781 [main] INFO org.apache.pig.Main - Logging error
messages to: /homes/viraj/pigscripts/pig_1253748381778.log
2009-09-23 23:26:21,963 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to
hadoop file system at: hdfs://localhost:8020
2009-09-23 23:26:22,227 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to
map-reduce job tracker at: localhost:50300
2009-09-23 23:26:27,187 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer
- Choosing to move algebraic foreach to combiner
2009-09-23 23:26:27,203 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2009-09-23 23:26:27,203 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2009-09-23 23:26:28,828 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2009-09-23 23:26:29,423 [Thread-9] WARN org.apache.hadoop.mapred.JobClient -
Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2009-09-23 23:26:29,478 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2009-09-23 23:27:29,828 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 50% complete
2009-09-23 23:27:59,764 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 50% complete
2009-09-23 23:28:57,249 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2009-09-23 23:28:57,249 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Successfully stored result in: "/user/viraj/finaloutput"
2009-09-23 23:28:57,267 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Records written : 60
2009-09-23 23:28:57,267 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Bytes written : 420
2009-09-23 23:28:57,267 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Success!
2009-09-23 23:28:57,367 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
2998: Unhandled internal error. File or directory '/user/viraj/finaloutput'
does not exist.
Details at logfile: /homes/viraj/pigscripts/pig_1253748381778.log
================================================================================
{code}
$shell> hadoop fs -ls /user/viraj/finaloutput
Found 1 items
-rw------- 3 viraj users 420 2009-09-23 23:42
/user/viraj/finaloutput/part-00000
{code}
================================================================================
Opening the log file:
================================================================================
Pig Stack Trace
---------------
ERROR 2998: Unhandled internal error. File or directory
'/user/viraj/finaloutput' does not exist.
java.io.IOException: File or directory '/user/viraj/finaloutput' does not exist.
at
org.apache.pig.tools.grunt.GruntParser.processMove(GruntParser.java:641)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:264)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:166)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:142)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
at org.apache.pig.Main.main(Main.java:397)
================================================================================
Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
