[
https://issues.apache.org/jira/browse/PIG-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614156#action_12614156
]
Yiping Han commented on PIG-318:
--------------------------------
A more general case which I raised in hadoop once is chaining M->R->M->R->...
by streaming the output of the reducers to the mappers of the next job. This
saves the round trip to disk and also, since the order of input to mappers
usually is not important, the second mapred job could start once the reducer of
the first job starts to output. This overlap could makes chaining jobs runs
even faster.
> Pipeline optimization for multiple reduces
> ------------------------------------------
>
> Key: PIG-318
> URL: https://issues.apache.org/jira/browse/PIG-318
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
>
> Any time we chain together M-R jobs, we doing it because we need separate
> reducer like with group by followed by order by. We don't really need the
> maps. The ideal graph for us would be:
>
> M->R->SortShuffle->R->SortShuffle->R ...
>
> This would allow us to save read from DFs and write to the local disk which
> could be fairly significant.
> Aparently this similar discussion took place on hadoop mailing list several
> times and this request was turned down. Main reason is that in their opinion
> cost of implementing something like that would outweigh the benefit.
> To make a persuasive case, we need to measure the overhead of the empty maps
> for "typical" queries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
