Hi folks, We are interested in adding a multi-user-unmanaged-am-pool to YARN, to be used by clients.
We see a pattern of services using YARN wanting to represent multiple users for efficiency reasons. Examples include Impala, Oozie, K8s-YARN integration. Today, Llama provides this functionality for Impala, but I am not sure if all these components would want/need to depend on Llama. I think adding this library code to YARN would help all of them and improve YARN's adoption. The proposal is to add library code that can be used to ask YARN for containers on behalf of different users. We can realize this by having a bunch of unmanaged AMs, one per queue/user, and have a threadpool handle the AM-RM interactions. I have included a prototype of this code, that is currently being used in OOZIE-1770. I think of this as an extension to AMRMClient* etc. that YARN currently has. Thoughts?
