[
https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated PIG-1917:
-------------------------------
Fix Version/s: (was: 0.8.0)
0.9.0
The change to parse "-Dmapred.child.java.opts='-Xmx1536m -Xss128k'" as one
argument is also going to break backward compatibility, if any existing pig
script is (accidentally) relying on that behavior. So making this change only
for 0.9.
Also, instead of unquoting all arguments, only the "-D" property arguments will
be unquoted. This change will only affect Native map-reduce statements (ie, not
streaming command).
> NativeMapReduce does not Allow Configuration Parameters containing Spaces
> --------------------------------------------------------------------------
>
> Key: PIG-1917
> URL: https://issues.apache.org/jira/browse/PIG-1917
> Project: Pig
> Issue Type: Bug
> Reporter: Mitesh Singh Jat
> Assignee: Thejas M Nair
> Fix For: 0.9.0
>
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain
> spaces.
> Considering a simple pig script with _wordcount_ as Native M-R job.
> {code:title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir'
> as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072
> -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because
> space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
> _-Xss128k"_ as the first argument to _wordcount_
> Even with other example M-R program, the parameter _mapred.child.java.opts_
> was assigned _"-Xmx1536m_ in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes.
> For example, here is the corresponding M-R Plan for the native M-R
> "MAPREDUCE" statement.
> {code}
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072
> -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
> --------
> {code}
> On executing above M-R job after copying 'input/WordCountInput.txt' into
> 'inputDir/', as Hadoop Job, this ran successfully.
> {code}
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072
> -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> {code}
> Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the
> *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
