[ https://issues.apache.org/jira/browse/IMPALA-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zach Amsden closed IMPALA-5920. ------------------------------- Resolution: Fixed Fix Version/s: Impala 2.11.0 IMPALA-5920: Remove admission control dependency on YARN RM jar Impala's admission controller relies on the YARN fair-scheduler.xml for configuration. That configuration is loaded using YARN directly (ie. as a library by the frontend). In Hadoop 3, a number of changes were made to the YARN resourcemanager which break Impala. While we eventually want to rethink the admission control configuration (IMPALA-4159), in the meantime we at least should avoid using unsupported YARN APIs. This patch removes the fe dependency on the YARN artifact 'hadoop-yarn-server-resourcemanager' which contains private APIs and isn't meant to be used as a library. A subset of the required code has been added in 'common/yarn-extras', taken from Hadoop 2.6.0 in CDH, e.g. see [1]. The code is added in packages 'org.apache.impala.*' instead of 'org.apache.yarn.*'. Some code could be copied as-is, those files are marked with the comment: //YARNUTIL: VANILLA Files that required some modifications are marked with: //YARNUTIL: MODIFIED Or, if all code except a dummy interface could be added: //YARNUTIL: DUMMY IMPL The goal the 'yarn-extras' is to make Impala's handling of the AC configuration self-sufficient, i.e. it shouldn't matter what version of Hadoop exists. As-is, this was tested and found to work when Hadoop 2.6 is installed. Because the yarn-extras/pom.xml still references hadoop-common, hadoop-yarn-common, and hadoop-yarn-api, additional testing will be required to ensure Impala using yarn-extras works when installed along side Hadoop 3. That testing for Hadoop 3 will be done later. Future changes will make any other changes required for existing code to work when Hadoop 3 is installed. Testing: * Ran existing tests on master. 1: https://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_vd_cdh_package_tarball_512.html Change-Id: I7efdd8ebea298836ca2a82c0a4ae037ac9285bcf Reviewed-on: http://gerrit.cloudera.org:8080/8035 Reviewed-by: Matthew Jacobs <mjac...@apache.org> Tested-by: Matthew Jacobs <mjac...@apache.org> > Remove admission control dependency on YARN resourcemanager > ----------------------------------------------------------- > > Key: IMPALA-5920 > URL: https://issues.apache.org/jira/browse/IMPALA-5920 > Project: IMPALA > Issue Type: Bug > Components: Frontend > Affects Versions: Impala 2.10.0 > Reporter: Matthew Jacobs > Assignee: Matthew Jacobs > Priority: Critical > Labels: admission-control, resource-management, rm > Fix For: Impala 2.11.0 > > > Impala's admission controller relies on the YARN fair-scheduler.xml for > configuration. That configuration is loaded using YARN directly (ie. as a > library by the frontend). In Hadoop 3, a number of changes were made to the > YARN resourcemanager which break Impala. While we eventually want to rethink > the admission control configuration (IMPALA-4159), in the meantime we at > least should avoid using unsupported YARN APIs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)