[
https://issues.apache.org/jira/browse/PIG-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Blue updated PIG-4850:
---------------------------
Status: Patch Available (was: Open)
> Registered jars do not use submit replication
> ---------------------------------------------
>
> Key: PIG-4850
> URL: https://issues.apache.org/jira/browse/PIG-4850
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Ryan Blue
> Assignee: Ryan Blue
> Attachments: PIG-4850.1.patch
>
>
> PIG-4074 added support for mapred.submit.replication, which sets the
> replication factor for files added to the distributed cache. The purpose is
> to avoid a huge number of task attempts downloading the same file in HDFS at
> once during localization and slowing down because of contention over few
> replicas. The replication factor for files was set correctly, but registered
> jars are added to HDFS through a different code path and weren't using the
> submit replication factor. This causes localization time for jobs to increase
> by as much as 10 minutes (at which point the tasks are killed).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
