Thanks for sharing! Aaron. Your comment is very helpful. Our end goal is to support multi-tenancy and also share the yarn cluster with MapReduce, Spark and other jobs. We probably need something else.
--- I wonder if there is any builtin functionalities in Flink or Yarn that already supports ACL on JobManager/AM. I also noticed is that flink doesn't have a way to set up ACLs for yarn containers so only the submitter or yarn admin can view the container logs. A related PR (https://github.com/apache/flink/pull/8760) was closed due to inactivity. How do people deal with container ACLs? Do most of flink-on-yarn dev/users use it without security? Or do we have to implement our own solution outside of flink/yarn? Please advise if anyone has any idea about this. Thanks very much! Best, Ethan On Mon, Mar 30, 2020 at 4:13 PM Aaron Langford <aaron.langfor...@gmail.com> wrote: > I'd be curious to see how others have done this, but our setup restricts > network access to machines in the YARN cluster to a jump box. Access to > Flink job manager is limited to whoever can ssh to that box, and that is > controlled with an Ansible playbook. Additionally, we have a list of users > specific to the cluster who can ssh to the machines in the YARN cluster > (also managed with Ansible). So the allowed users are the intersection of > the jump server ACL and the YARN cluster ACL. Web access happens by using a > local socks proxy along with the Foxy Proxy browser plugin. It's definitely > pretty crude and doesn't scale super well as more teams need varying access > policies to different YARN clusters/jobs, but it is satisfying our needs > for now. One big simplifying assumption is that we don't support shared > clusters. Amazon's EMR service allows teams to spin up clusters really > easily, so we can get away with saying that the machine network rules can > map to the actual access control rules any given job/team might need. > > Aaron > > On Mon, Mar 30, 2020 at 12:53 PM Ethan Li <ethanopensou...@gmail.com> > wrote: > >> Hi Team, >> >> I am evaluating Flink on yarn. I can submit a flink job to a secured Yarn >> cluster and the job can run correctly. But flink jobmanager UI seems >> accessibly by everyone. Is there anyway in Flink or Yarn to secure it with >> ACLs? >> >> Thanks, >> Ethan >> >
