[
https://issues.apache.org/jira/browse/PIG-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-2779:
-----------------------------
Attachment: PIG-2779.5.patch
Agreed, let's hold off on feature creep. I've made a few edits and updated the
patch. A few things to note:
- I added more assertions for all parallel values after each test in
{{TestNumberOfReducers}} and {{TestJobSubmission}}.
- I changed the contract of {{PigReducerEstimator}} to return -1 if estimation
can't be done (i.e. HBaseStorage), since returning 1 was misleading.
- I refactored some common code in {{TestNumberOfReducers}}.
Take a look and let me know what you think Jie. I'll run the test suite with
our CI server tonight.
> Refactoring the code for setting number of reducers
> ---------------------------------------------------
>
> Key: PIG-2779
> URL: https://issues.apache.org/jira/browse/PIG-2779
> Project: Pig
> Issue Type: Bug
> Reporter: Jie Li
> Assignee: Jie Li
> Fix For: 0.11
>
> Attachments: PIG-2779.0.patch, PIG-2779.1.patch, PIG-2779.2.patch,
> PIG-2779.3.patch, PIG-2779.4.patch, PIG-2779.5.patch,
> TestNumberOfReducers.java, TestNumberOfReducers.java
>
>
> As PIG-2652 observed, currently the code for setting number of reducers is a
> little messy. MapReduceOper.requestedParallelism seems being misused in some
> plases, and now we support runtime estimation of #reducer which further
> complicates the problem.
> For example, if we specify parallel 1 for the order-by, the estimated
> #reducer will be used. If we specify parallel 2 while it estimates 4,
> order-by will fail due to "Illegal partition for Null". If we specify
> parallel 4 while it estimates 2, then some reducers will have nothing to do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
