[
https://issues.apache.org/jira/browse/MAPREDUCE-7548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jose Luis López updated MAPREDUCE-7548:
---------------------------------------
Description:
*Problem*: on a Kerberos cluster with service-level authorization
({{hadoop.security.authorization=true}}), every MapReduce job fails. Each
task's connection to its application master is refused, and the task exits:
{noformat}
WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child :
java.io.UncheckedIOException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
Protocol interface org.apache.hadoop.mapred.TaskUmbilicalProtocol is not known.
{noformat}
*Cause*: the protocol name and the ACL key do not match.
* {{TaskUmbilicalProtocolPB}} declares {{@ProtocolInfo(protocolName =
"org.apache.hadoop.mapred.TaskUmbilicalProtocol")}}, so the RPC server resolves
an incoming call to the {{TaskUmbilicalProtocol}} interface and authorizes it
against that class.
* {{MRAMPolicyProvider}} registers {{security.job.task.protocol.acl}} under
{{TaskUmbilicalProtocolPB.class}}, so {{ServiceAuthorizationManager}} has no
ACL for {{TaskUmbilicalProtocol}} and rejects the call.
HDFS uses the other convention, which works: {{HDFSPolicyProvider}} lists
{{ClientProtocol.class}} for {{ClientNamenodeProtocolPB}}, whose protocol name
is {{org.apache.hadoop.hdfs.protocol.ClientProtocol}}. {{MRClientProtocolPB}}
is consistent too, since its protocol name is its own class name.
*To reproduce*: a secure single-node cluster (Kerberos,
{{hadoop.security.authorization=true}}), then any job, e.g. {{yarn jar
hadoop-mapreduce-examples-*.jar pi 2 100}}. All task attempts fail with the
exception above. Seen on trunk (3.6.0-SNAPSHOT).
Found during the Jetty 12 (EE8) migration testing (HADOOP-19912), by an
end-to-end test of a secure single-node cluster built from trunk. The bug is on
trunk itself; the migration does not cause it.
*Proposed fix*: register the ACL under the class the server authorizes against.
In {{MRAMPolicyProvider}}:
{code:java}
new Service(
MRJobConfig.MR_AM_SECURITY_SERVICE_AUTHORIZATION_TASK_UMBILICAL,
TaskUmbilicalProtocol.class), // was TaskUmbilicalProtocolPB.class
{code}
{{Server.authorize()}} turns the protocol name from the connection header into
a class with {{conf.getClassByName(protocolName)}}, which is
{{org.apache.hadoop.mapred.TaskUmbilicalProtocol}} (in
hadoop-mapreduce-client-core, already a dependency of
hadoop-mapreduce-client-app). The ACL key ({{security.job.task.protocol.acl}})
and the wire protocol stay the same, so nothing changes for clients or
configuration. The {{MRClientProtocolPB}} entry is already right.
*Test*: a unit test in hadoop-mapreduce-client-app that starts
{{TaskAttemptListenerImpl}} with {{hadoop.security.authorization=true}} and
calls it through {{TaskUmbilicalProtocolPBClientImpl}}; it fails today with the
exception above. A cheaper guard against the same mistake: for each PB
interface the AM serves, check that {{MRAMPolicyProvider}} has a service for
the class named by its {{@ProtocolInfo(protocolName)}}.
was:
*Problem*: on a Kerberos cluster with service-level authorization
({{hadoop.security.authorization=true}}), every MapReduce job fails. Each
task's connection to its application master is refused, and the task exits:
{noformat}
WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child :
java.io.UncheckedIOException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
Protocol interface org.apache.hadoop.mapred.TaskUmbilicalProtocol is not known.
{noformat}
*Cause*: the protocol name and the ACL key do not match.
* {{TaskUmbilicalProtocolPB}} declares {{@ProtocolInfo(protocolName =
"org.apache.hadoop.mapred.TaskUmbilicalProtocol")}}, so the RPC server resolves
an incoming call to the {{TaskUmbilicalProtocol}} interface and authorizes it
against that class.
* {{MRAMPolicyProvider}} registers {{security.job.task.protocol.acl}} under
{{TaskUmbilicalProtocolPB.class}}, so {{ServiceAuthorizationManager}} has no
ACL for {{TaskUmbilicalProtocol}} and rejects the call.
HDFS uses the other convention, which works: {{HDFSPolicyProvider}} lists
{{ClientProtocol.class}} for {{ClientNamenodeProtocolPB}}, whose protocol name
is {{org.apache.hadoop.hdfs.protocol.ClientProtocol}}. {{MRClientProtocolPB}}
is consistent too, since its protocol name is its own class name.
*To reproduce*: a secure single-node cluster (Kerberos,
{{hadoop.security.authorization=true}}), then any job, e.g. {{yarn jar
hadoop-mapreduce-examples-*.jar pi 2 100}}. All task attempts fail with the
exception above. Seen on trunk (3.6.0-SNAPSHOT).
Found by an end-to-end test of a secure single-node cluster built from trunk.
*Proposed fix*: register the ACL under the class the server authorizes against.
In {{MRAMPolicyProvider}}:
{code:java}
new Service(
MRJobConfig.MR_AM_SECURITY_SERVICE_AUTHORIZATION_TASK_UMBILICAL,
TaskUmbilicalProtocol.class), // was TaskUmbilicalProtocolPB.class
{code}
{{Server.authorize()}} turns the protocol name from the connection header into
a class with {{conf.getClassByName(protocolName)}}, which is
{{org.apache.hadoop.mapred.TaskUmbilicalProtocol}} (in
hadoop-mapreduce-client-core, already a dependency of
hadoop-mapreduce-client-app). The ACL key ({{security.job.task.protocol.acl}})
and the wire protocol stay the same, so nothing changes for clients or
configuration. The {{MRClientProtocolPB}} entry is already right.
*Test*: a unit test in hadoop-mapreduce-client-app that starts
{{TaskAttemptListenerImpl}} with {{hadoop.security.authorization=true}} and
calls it through {{TaskUmbilicalProtocolPBClientImpl}}; it fails today with the
exception above. A cheaper guard against the same mistake: for each PB
interface the AM serves, check that {{MRAMPolicyProvider}} has a service for
the class named by its {{@ProtocolInfo(protocolName)}}.
> MapReduce tasks are rejected by the AM when hadoop.security.authorization is
> enabled: TaskUmbilicalProtocol is not known
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-7548
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7548
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, security
> Affects Versions: 3.6.0
> Reporter: Jose Luis López
> Priority: Major
>
> *Problem*: on a Kerberos cluster with service-level authorization
> ({{hadoop.security.authorization=true}}), every MapReduce job fails. Each
> task's connection to its application master is refused, and the task exits:
> {noformat}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child :
> java.io.UncheckedIOException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> Protocol interface org.apache.hadoop.mapred.TaskUmbilicalProtocol is not
> known.
> {noformat}
> *Cause*: the protocol name and the ACL key do not match.
> * {{TaskUmbilicalProtocolPB}} declares {{@ProtocolInfo(protocolName =
> "org.apache.hadoop.mapred.TaskUmbilicalProtocol")}}, so the RPC server
> resolves an incoming call to the {{TaskUmbilicalProtocol}} interface and
> authorizes it against that class.
> * {{MRAMPolicyProvider}} registers {{security.job.task.protocol.acl}} under
> {{TaskUmbilicalProtocolPB.class}}, so {{ServiceAuthorizationManager}} has no
> ACL for {{TaskUmbilicalProtocol}} and rejects the call.
> HDFS uses the other convention, which works: {{HDFSPolicyProvider}} lists
> {{ClientProtocol.class}} for {{ClientNamenodeProtocolPB}}, whose protocol
> name is {{org.apache.hadoop.hdfs.protocol.ClientProtocol}}.
> {{MRClientProtocolPB}} is consistent too, since its protocol name is its own
> class name.
> *To reproduce*: a secure single-node cluster (Kerberos,
> {{hadoop.security.authorization=true}}), then any job, e.g. {{yarn jar
> hadoop-mapreduce-examples-*.jar pi 2 100}}. All task attempts fail with the
> exception above. Seen on trunk (3.6.0-SNAPSHOT).
> Found during the Jetty 12 (EE8) migration testing (HADOOP-19912), by an
> end-to-end test of a secure single-node cluster built from trunk. The bug is
> on trunk itself; the migration does not cause it.
> *Proposed fix*: register the ACL under the class the server authorizes
> against. In {{MRAMPolicyProvider}}:
> {code:java}
> new Service(
> MRJobConfig.MR_AM_SECURITY_SERVICE_AUTHORIZATION_TASK_UMBILICAL,
> TaskUmbilicalProtocol.class), // was TaskUmbilicalProtocolPB.class
> {code}
> {{Server.authorize()}} turns the protocol name from the connection header
> into a class with {{conf.getClassByName(protocolName)}}, which is
> {{org.apache.hadoop.mapred.TaskUmbilicalProtocol}} (in
> hadoop-mapreduce-client-core, already a dependency of
> hadoop-mapreduce-client-app). The ACL key
> ({{security.job.task.protocol.acl}}) and the wire protocol stay the same, so
> nothing changes for clients or configuration. The {{MRClientProtocolPB}}
> entry is already right.
> *Test*: a unit test in hadoop-mapreduce-client-app that starts
> {{TaskAttemptListenerImpl}} with {{hadoop.security.authorization=true}} and
> calls it through {{TaskUmbilicalProtocolPBClientImpl}}; it fails today with
> the exception above. A cheaper guard against the same mistake: for each PB
> interface the AM serves, check that {{MRAMPolicyProvider}} has a service for
> the class named by its {{@ProtocolInfo(protocolName)}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
