Hey Sergiy, I think a similar approach IIUC, where an AM for a app running on a cluster acts as an unmanaged AM on another cluster. I believe they use a separate UGI for each sub-cluster and wrap it around a doAs before the actual allocate call.
Subru might be able to give more details. Cheers -Arun On Thu, Mar 16, 2017 at 2:34 PM, Jason Lowe <jl...@yahoo-inc.com.invalid> wrote: > The doAs method in UserGroupInformation is what you want when dealing with > multiple UGIs. It determines what UGI instance the code within the doAs > scope gets when that code tries to lookup the current user. > Each AM is designed to run in a separate JVM, so each has some main()-like > entry point that does everything to setup the AM. Theoretically all you > need to do is create two, separate UGIs then use each instance to perform a > doAs wrapping the invocation of the corresponding AM's entry point. After > that, everything that AM does will get the UGI of the doAs invocation as > the current user. Since the AMs are running in separate doAs instances > they will get separate UGIs for the current user and thus separate > credentials. > Jason > > > On Thursday, March 16, 2017 4:03 PM, Sergiy Matusevych < > sergiy.matusev...@gmail.com> wrote: > > > Hi Jason, > > Thanks a lot for your help again! Having two separate UserGroupInformation > instances is exactly what I had in mind. What I do not understand, though, > is how to make sure that our second call to .regsiterApplicationMaster() > will pick the right UserGroupInformation object. I would love to find a way > that does not involve any changes to the YARN client, but if we have to > patch it, of course, I agree that we need to have a generic yet minimally > invasive solution. > Thank you!Sergiy. > > > On Thu, Mar 16, 2017 at 8:03 AM, Jason Lowe <jl...@yahoo-inc.com> wrote: > > > > I believe a cleaner way to solve this problem is to create two, > _separate_ UserGroupInformation objects and wrap each AM instances in a UGI > doAs so they aren't trying to share the same credentials. This is one > example of a token bleeding over and causing problems. I suspect trying to > fix these one-by-one as they pop up is going to be frustrating compared to > just ensuring the credentials remain separate as if they really were > running in separate JVMs. > > > > Adding Daryn who knows a lot more about the UGI stuff so he can correct > any misunderstandings on my part. > > > > Jason > > > > > > On Wednesday, March 15, 2017 1:11 AM, Sergiy Matusevych < > sergiy.matusev...@gmail.com> wrote: > > > > > > Hi YARN developers, > > > > I have an interesting problem that I think is related to YARN Java > client. > > I am trying to launch *two* application masters in one container. To be > > more specific, I am starting a Spark job on YARN, and launch an Apache > REEF > > Unmanaged AM from the Spark Driver. > > > > Technically, YARN Resource Manager should not care which process each AM > > runs in. However, there is a problem with the YARN Java client > > implementation: there is a global UserGroupInformation object that holds > > the user credentials of the current RM session. This data structure is > > shared by all AMs, and when REEF application tries to register the second > > (unmanaged) AM, the client library presents to YARN RM all credentials, > > including the security token of the first (managed) AM. YARN rejects such > > registration request, throwing InvalidApplicationMasterRequestException > > "Application Master is already registered". > > > > I feel like this issue can be resolved by a relatively small update to > the > > YARN Java client - e.g. by introducing a new variant of the > > AMRMClientAsync.registerApplicationMaster() that would take the required > > security token (instead of getting it implicitly from the > > UserGroupInformation.getCurrentUser().getCredentials() etc.), or having > > some sort of RM session class that would wrap all data that is currently > > global. I need to think about the elegant API for it. > > > > What do you guys think? I would love to work on this problem and send > you a > > pull request for the upcoming 2.9 release. > > > > Cheers, > > Sergiy. > > > > > > > >
