[
https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HADOOP-2131:
----------------------------------
Status: Open (was: Patch Available)
Please go ahead and deprecate the old {{mapred.speculative.execution}} config
in favour of the new ones which should be set to *true* in hadoop-default.xml.
For 0.16.0 we should let {{mapred.speculative.execution}} over-ride the new
ones since it means folks actually went ahead and cared about the non-default
value and hence set it in their hadoop-site.xml.
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit
> the data. The reducers in this are identity reducers. The data processed by
> these jobs is huge. There could be slow nodes in this cluster and some of the
> reducers run twice as slow as their counterparts. This could result in a long
> tail. Speculative execution would help greatly in such cases. However given
> the current hadoop, we have to select speculative execution for both maps and
> reducers. In this case hurting the map performance as they are fetching data
> from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this
> problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
