[
https://issues.apache.org/jira/browse/TEZ-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194564#comment-14194564
]
Rajesh Balamohan commented on TEZ-1731:
---------------------------------------
lgtm. Minor comment
{code}
t1mergeManager.onDiskMerger.merge(t0MergeFiles); <== I believe you meant to add
"t1MergeFiles".
{code}
> OnDiskMerger can end up clobbering files across tasks with LocalDiskFetch
> -------------------------------------------------------------------------
>
> Key: TEZ-1731
> URL: https://issues.apache.org/jira/browse/TEZ-1731
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: TEZ-1731.1.txt
>
>
> When an on disk fetch starts with LOCAL files (optimize.local.fetch), the
> filename used by the merger is based on the source file name. This name can
> be the same for all tasks reading the same input on the node - and can result
> in files being overwritten between tasks, depending on the order in which
> events are processed, and the dir allocated by the local dir-allocator.
> Leads to ChecksumExceptions, and FileNotFoundExceptions during the merge.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
