[jira] [Commented] (FLINK-3670) Kerberos: Improving long-running streaming jobs
[ https://issues.apache.org/jira/browse/FLINK-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289619#comment-15289619 ] Eron Wright commented on FLINK-3670: - After some design discussion, the keytab approach will be used as outlined in FLINK-3929. > Kerberos: Improving long-running streaming jobs > --- > > Key: FLINK-3670 > URL: https://issues.apache.org/jira/browse/FLINK-3670 > Project: Flink > Issue Type: Improvement > Components: Command-line client, Local Runtime >Reporter: Maximilian Michels >Assignee: Eron Wright > > We have seen in the past, that Hadoop's delegation tokens are subject to a > number of subtle token renewal bugs. In addition, they have a maximum life > time that can be worked around but is very inconvenient for the user. > As per [mailing list > discussion|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Kerberos-for-Streaming-amp-Kafka-td10906.html], > a way to work around the maximum life time of DelegationTokens would be to > pass the Kerberos principal and key tab upon job submission. A daemon could > then periodically renew the ticket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3670) Kerberos: Improving long-running streaming jobs
[ https://issues.apache.org/jira/browse/FLINK-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232054#comment-15232054 ] Niels Basjes commented on FLINK-3670: - A while ago I found that part of the problem is in the upstream tools that are used. See a similar bug report for Spark (SPARK-11182) and what looks like an important blocker to really fix this HDFS-9276 > Kerberos: Improving long-running streaming jobs > --- > > Key: FLINK-3670 > URL: https://issues.apache.org/jira/browse/FLINK-3670 > Project: Flink > Issue Type: Improvement > Components: Command-line client, Local Runtime >Reporter: Maximilian Michels > > We have seen in the past, that Hadoop's delegation tokens are subject to a > number of subtle token renewal bugs. In addition, they have a maximum life > time that can be worked around but is very inconvenient for the user. > As per [mailing list > discussion|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Kerberos-for-Streaming-amp-Kafka-td10906.html], > a way to work around the maximum life time of DelegationTokens would be to > pass the Kerberos principal and key tab upon job submission. A daemon could > then periodically renew the ticket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3670) Kerberos: Improving long-running streaming jobs
[ https://issues.apache.org/jira/browse/FLINK-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217092#comment-15217092 ] Eron Wright commented on FLINK-3670: - Another possibility worth considering is to leverage Hadoop's 'proxy user' functionality. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html In this approach, the JobManager impersonates the job submitter when accessing HDFS, HBASE, or Hive. Those servers would be configured to treat the JobManager principal as a proxy user. Note that the above solution isn't general, since Kafka (for example) doesn't provide proxy user functionality.Maybe both options could be provided. > Kerberos: Improving long-running streaming jobs > --- > > Key: FLINK-3670 > URL: https://issues.apache.org/jira/browse/FLINK-3670 > Project: Flink > Issue Type: Improvement > Components: Command-line client, Local Runtime >Reporter: Maximilian Michels > > We have seen in the past, that Hadoop's delegation tokens are subject to a > number of subtle token renewal bugs. In addition, they have a maximum life > time that can be worked around but is very inconvenient for the user. > As per [mailing list > discussion|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Kerberos-for-Streaming-amp-Kafka-td10906.html], > a way to work around the maximum life time of DelegationTokens would be to > pass the Kerberos principal and key tab upon job submission. A daemon could > then periodically renew the ticket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
