Lucas Lemanowicz created PIG-2728:
-------------------------------------
Summary: Grunt shell automagically sets number of reducers and
doesn't allow you to override it
Key: PIG-2728
URL: https://issues.apache.org/jira/browse/PIG-2728
Project: Pig
Issue Type: Bug
Components: grunt
Affects Versions: 0.8.1
Environment: CentOS
12 boxes
30 map/reduce tasks max per box @ 2GB memory each
Reporter: Lucas Lemanowicz
When running Pig scripts through the grunt shell I kept getting Java Heap Space
error messages. The cause was that my Hadoop cluster was running 467 map tasks
which were being passed down to only 199 reducers. The notice I was getting
through the Grunt shell was:
"Neither PARALLEL nor default parallelism is set for this job. Setting number
of reducers to 199"
I tried the following commands, none of which worked:
SET DEFAULT_PARALLELISM 2;
SET mapred.reduce.tasks 450;
SET mapred.running.reduce.limit 30;
When I ran the script directly from a file (pig filename.pig) it ran without
problems.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
