[
https://issues.apache.org/jira/browse/PIG-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264151#comment-13264151
]
Dmitriy V. Ryaboy commented on PIG-2675:
----------------------------------------
We used to do this at compile-time, but this caused problems with reducer
estimation, which happens at run-time -- either we have to not estimate
reducers after compilation, or we have to punt on the extra limit job decision
until the reducer estimation happens. Clearly, we should do the latter.
Implementation is likely to create a visitor that reverts the plan changes
LimitAdjuster creates (moves the proper storers back to the first job, etc).
> Optimization: Remove unnecessary Limit jobs from plan
> -----------------------------------------------------
>
> Key: PIG-2675
> URL: https://issues.apache.org/jira/browse/PIG-2675
> Project: Pig
> Issue Type: Improvement
> Reporter: Dmitriy V. Ryaboy
> Assignee: Dmitriy V. Ryaboy
>
> LIMIT operator always inserts a limiting single-reducer job after PIG-2652.
> We can optimize this job away when the preceding job only has 1 reducer at
> run-time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
