[ https://issues.apache.org/jira/browse/HIVE-17781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211391#comment-16211391 ]
Mithun Radhakrishnan commented on HIVE-17781: --------------------------------------------- +1. Tests pass. Rebasing before commit. > Map MR settings to Tez settings via DeprecatedKeys > -------------------------------------------------- > > Key: HIVE-17781 > URL: https://issues.apache.org/jira/browse/HIVE-17781 > Project: Hive > Issue Type: Bug > Components: Configuration, Tez > Affects Versions: 3.0.0 > Reporter: Mithun Radhakrishnan > Assignee: Chris Drome > Attachments: HIVE-17781.1.patch > > > Here's one that [~cdrome] and [~thiruvel] worked on: > We found that certain Hadoop Map/Reduce settings that are set in site config > files do not take effect in Hive jobs, because the Tez site configs do not > contain the same settings. > In Yahoo's case, the problem was that, at the time, there was no mapping > between {{MRJobConfig.COMPLETED_MAPS_FOR_REDUCE_SLOWSTART}} and > {{TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION}}. There were situations where > significant capacity on production clusters were being used up doing nothing, > while waiting for slow tasks to complete. This would have been avoided, were > the mappings in place. > Tez provides a {{DeprecatedKeys}} utility class, to help map MR settings to > Tez settings. Hive should use this to ensure that the mappings are in sync. > (Note to self: YHIVE-883) -- This message was sent by Atlassian JIRA (v6.4.14#64029)