[
https://issues.apache.org/jira/browse/PIG-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897451#action_12897451
]
Richard Ding commented on PIG-1458:
-----------------------------------
The proposal is to run another map-reduce job to merge the small files before
the replicated join. This additional job will be added to the MR plan at the
compile time.
We consider three cases of a replicated join:
# The right input is a map-only job and input files exist at the compile time.
# The right input is a map-only job and input files do not exist at the compile
time.
# The right input is a map-reduce job.
For 1., if the number of files exceeds the threshold specified in the property
file (_pig.frjoin.merge.files.threshold_), a merge job is added between right
input job and FR join job.
For 3., if the number of reducers exceeds the threshold specified in the
property file (_pig.frjoin.merge.files.threshold_), a merge job is added
between right input job and FR join job.
For 2., if the flag specified in the property file
(_pig.frjoin.merge.files.optimistic_) is false, a merge job is added between
right input job and FR join job. The default value of this flag is false.
> aggregate files for replicated join
> -----------------------------------
>
> Key: PIG-1458
> URL: https://issues.apache.org/jira/browse/PIG-1458
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Richard Ding
> Fix For: 0.8.0
>
>
> We have noticed that if the smaller data in replicated join has many files,
> this puts unneeded burden on the name node. pre-aggregating the files can
> improve the situation
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
