I agree that it is reasonable to use Hadoop DTs as you describe. That approach is even recommended in YARN's documentation (see Securing Long-lived YARN Services on the YARN Application Security page). But one of the goals of Kerberos integration is to support Kerberized data access for connectors other than HDFS, such as Kafka, Cassandra, and Elasticsearch. So your second point makes sense too, suggesting a general architecture for managing secrets (DTs, keytabs, certificates, oauth tokens, etc.) within the cluster.
There's quite a few aspects to Flink security, including: 1. data access (e.g. how a connector authenticates to a data source) 2. service authorization and network security (e.g. how a Flink cluster protects itself from unauthorized access) 3. multi-user support (e.g. multi-user Flink clusters, RBAC) I mention these aspects to clarify your point about AuthN, which I took to be related to (1). Do tell if I misunderstood. Eron On Wed, Dec 20, 2017 at 11:21 AM, Shuyi Chen <suez1...@gmail.com> wrote: > Hi community, > > We are working on secure Flink on YARN. The current Flink-Yarn-Kerberos > integration will require each container of a job to log in Kerberos via > keytab every say, 24 hours, and does not use any Hadoop delegation token > mechanism except when localizing the container. As I fixed the current > Flink-Yarn-Kerberos (FLINK-8275) and tried to add more > features(FLINK-7860), I have some concern regarding the current > implementation. It can pose a scalability issue to the KDC, e.g., if YARN > cluster is restarted and all 10s of thousands of containers suddenly DDOS > KDC. > > I would like to propose to improve the current Flink-YARN-Kerberos > integration by doing something like the following: > 1) AppMaster (JobManager) periodically authenticate the KDC, get all > required DTs for the job. > 2) all other TM or TE containers periodically retrieve new DTs from the > AppMaster (either through a secure HDFS folder, or a secure Akka channel) > > Also, we want to extend Flink to support pluggable AuthN mechanism, because > we have our own internal AuthN mechanism. We would like add support in > Flink to authenticate periodically to our internal AuthN service as well > through, e.g., dynamic class loading, and use similar mechanism to > distribute the credential from the appMaster to containers. > > I would like to get comments and feedbacks. I can also write a design doc > or create a Flip if needed. Thanks a lot. > > Shuyi > > > > -- > "So you have to trust that the dots will somehow connect in your future." >
