[
https://issues.apache.org/jira/browse/PIG-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409200#comment-13409200
]
Bill Graham commented on PIG-2779:
----------------------------------
+1 for the suggested refactor, as it would be good to clear up distinctions
between default parallel, requested parallelism and runtime parallelism.
Regarding the patch, you're changing logic that was added in the particularly
nasty PIG-2652. Do all the tests in the comments and final patch from PIG-2652
pass?
> Refactoring the code for setting number of reducers
> ---------------------------------------------------
>
> Key: PIG-2779
> URL: https://issues.apache.org/jira/browse/PIG-2779
> Project: Pig
> Issue Type: Bug
> Reporter: Jie Li
> Assignee: Jie Li
> Fix For: 0.11
>
> Attachments: PIG-2779.0.patch, TestNumberOfReducers.java,
> TestNumberOfReducers.java
>
>
> As PIG-2652 observed, currently the code for setting number of reducers is a
> little messy. MapReduceOper.requestedParallelism seems being misused in some
> plases, and now we support runtime estimation of #reducer which further
> complicates the problem.
> For example, if we specify parallel 1 for the order-by, the estimated
> #reducer will be used. If we specify parallel 2 while it estimates 4,
> order-by will fail due to "Illegal partition for Null". If we specify
> parallel 4 while it estimates 2, then some reducers will have nothing to do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
