[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-2131: ---------------------------------- Status: Open (was: Patch Available) Please go ahead and deprecate the old {{mapred.speculative.execution}} config in favour of the new ones which should be set to *true* in hadoop-default.xml. For 0.16.0 we should let {{mapred.speculative.execution}} over-ride the new ones since it means folks actually went ahead and cared about the non-default value and hence set it in their hadoop-site.xml. > Speculative execution should be allowed for reducers only > --------------------------------------------------------- > > Key: HADOOP-2131 > URL: https://issues.apache.org/jira/browse/HADOOP-2131 > Project: Hadoop > Issue Type: Improvement > Components: mapred > Environment: Hadoop job, map fetches data from external systems > Reporter: Srikanth Kakani > Assignee: Amareshwari Sri Ramadasu > Priority: Critical > Fix For: 0.16.0 > > Attachments: patch-2131.txt > > > Consider hadoop jobs where maps fetch data from external systems, and emit > the data. The reducers in this are identity reducers. The data processed by > these jobs is huge. There could be slow nodes in this cluster and some of the > reducers run twice as slow as their counterparts. This could result in a long > tail. Speculative execution would help greatly in such cases. However given > the current hadoop, we have to select speculative execution for both maps and > reducers. In this case hurting the map performance as they are fetching data > from external systems thereby overloading the external systems. > Speculative execution only on reducers would be a great way to solve this > problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.