[jira] [Commented] (FLINK-11088) Improve Kerberos Authentication using Keytab in YARN proxy user mode
[ https://issues.apache.org/jira/browse/FLINK-11088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721857#comment-16721857 ] Rong Rong commented on FLINK-11088: --- I further dig into the details on the document on Hadoop side and seems like there are 3 recommended way of distributing credentials to secure long running service on YARN. See here: https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Securing_Long-lived_YARN_Services. I am not sure whether this applies to other cluster resource management system, but I think it is worthwhile to take a look. For one: the current way of letting all JM and TMs renews keytab with KDC seems to be a problem. If we can have AM or JM renewing with keytab credential and distribute them via delegation token to all TMs it will release lots of loads on KDC server. I will start drafting a simple discussion doc if the community thinks this is worth to dig deeper. Any thoughts [~till.rohrmann] [~aljoscha] ? > Improve Kerberos Authentication using Keytab in YARN proxy user mode > > > Key: FLINK-11088 > URL: https://issues.apache.org/jira/browse/FLINK-11088 > Project: Flink > Issue Type: Improvement > Components: Security, YARN >Reporter: Rong Rong >Assignee: Rong Rong >Priority: Major > > Currently flink-yarn assumes keytab is shipped as application master > environment local resource on client side and will be distributed to all the > TMs. This does not work for YARN proxy user mode [1] since proxy user or > super user might not have access to actual users' keytab, but can request > delegation tokens on users' behalf. > Based on the type of security options for long-living YARN service[2], we > propose to have the keytab file path discovery configurable depending on the > launch mode of the YARN client. > Reference: > [1] > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html > [2] > https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Securing_Long-lived_YARN_Services -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-11088) Improve Kerberos Authentication using Keytab in YARN proxy user mode
[ https://issues.apache.org/jira/browse/FLINK-11088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714128#comment-16714128 ] Rong Rong commented on FLINK-11088: --- Initial investigation needs to find a way to distinguish between the two types of authentication method: Keytab and Delegation token. However, since delegation tokens normally expires within a week, see: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/security-kerberos.html#using-kinit-yarn-only, there should've been a configurable API to specify the way to pass over the Kerberos keytab to YARN application master. The proposal consists of several combination of scenarios: 1. Delegation token only - Cluster is short living. No keytab file 2. Delegation token on launch - Cluster can be long living if keytab file was supplied, or keytab acquisition method is defined. 3. Keytab on launch - Cluster is long living, Keytab is passed as YARN local resource (current method) Please comment if you think there's any other ways of authenticating Flink app. > Improve Kerberos Authentication using Keytab in YARN proxy user mode > > > Key: FLINK-11088 > URL: https://issues.apache.org/jira/browse/FLINK-11088 > Project: Flink > Issue Type: Improvement > Components: Security, YARN >Reporter: Rong Rong >Assignee: Rong Rong >Priority: Major > > Currently flink-yarn assumes keytab is shipped as application master > environment local resource on client side and will be distributed to all the > TMs. This does not work for YARN proxy user mode since proxy user or super > user does not have access to actual user's keytab but only delegation tokens. > We propose to have the keytab file path discovery configurable depending on > the launch mode of the YARN client. > Reference: > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
