[
https://issues.apache.org/jira/browse/PIG-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805919#comment-13805919
]
Aniket Mokashi commented on PIG-3047:
-------------------------------------
Your patch does not have the right fix for this problem. If you look closely,
the first fragment of replFiles is not added to distributed cache and hence we
should not add all of replFiles. The reason first patch works is because at the
time this check is made replFiles[i] is null hence it is skipped. Also, since
you are not recursively listing the files, you do not hit the problem we hit in
the second patch (hidden _logs directory). The reason second patch test case
shows big numbers is because of hidden _logs directory in the replicated data
path. I fixed the Utils.getPathLength method to ignore the hidden files. *This
also affects reducer estimation code path by ignoring hidden directories while
listing input*, which I feel is right way to go.
> Check the size of a relation before adding it to distributed cache in
> Replicated join
> -------------------------------------------------------------------------------------
>
> Key: PIG-3047
> URL: https://issues.apache.org/jira/browse/PIG-3047
> Project: Pig
> Issue Type: Improvement
> Reporter: Julien Le Dem
> Assignee: Ido Hadanny
> Fix For: 0.12.1
>
> Attachments: fix_pig_3047_1.diff, fix_pig_3047.diff
>
>
> Right now if someone makes a mistake and put the large relation last, Pig
> will copy a huge file into distributed cache and it will take a long time
> before the job eventually fails. It would be better to check before copying
> the relation that it is of reasonable size.
> <1 GB ?
--
This message was sent by Atlassian JIRA
(v6.1#6144)
