[
https://issues.apache.org/jira/browse/PIG-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903892#action_12903892
]
Thejas M Nair commented on PIG-1458:
------------------------------------
+1
Looks good. Some minor comments -
- If the preceding op is a native MR job (for native mapreduce operator), we
don't know how many reducers will be run , pig should use the
pig.frjoin.merge.files.optimistic property in that case. For native mr job, the
map plan will be empty, so currently the check for number of roots will return
false.
- If one input file has been found to have several files, we can stop there,
and avoid checking other files.
{code}
} else if (!frJoinOptimisticFileMerge) {
// file doesn't exist yet. Treat it as having too many
// files
numFiles = frJoinFileMergeThreshold;
}
{code}
{code}
} else if (!frJoinOptimisticFileMerge) {
// file doesn't exist yet. Treat it as having too many
// files
return true;
}
{code}
> aggregate files for replicated join
> -----------------------------------
>
> Key: PIG-1458
> URL: https://issues.apache.org/jira/browse/PIG-1458
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Richard Ding
> Fix For: 0.8.0
>
> Attachments: PIG-1458.patch
>
>
> We have noticed that if the smaller data in replicated join has many files,
> this puts unneeded burden on the name node. pre-aggregating the files can
> improve the situation
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
