I'm doing precisely the opposite. My own node manager (MY_NM) is an AM in YARN and, therefore, each MY_NM is expected to run inside a YARN container. What I trying to do is to execute the AM (MY_NM) on each slave.
For the reason, I need to launch an AM on a specific node but Hadoop-2.6.0 ignores what I describe in a ResourceRequest object. According to what I've understood, RMAppManager in ResourceManager creates a RMAppImpl object for each application and a RMAppAttempt object for each application attempt. When the state of RMAppAttempt changes from SUBMITTED to SCHEDULED, RMAppAttempt invalidates necessary information to launch a container for AM such as # containers, priority, resource name (=hostname), whether to relax locality as in the above code and then asks YarnScheduler to allocate a container. I conclude that any application master cannot be launched on a specific node in Hadoop-2.6.0. As a workaround, I try to execute an AM directly on a specific node from the command line without submitting it to YARN and register the AM to RM. It however fails (of course :P) because the AM is not given a delegation token necessary to communicate with a RPC server in RM. Anyway I need to find another way of doing that or I have to alter the design of my framework. Thanks. - Dongwon Kim 2015-04-01 17:50 GMT+09:00 Drake민영근 <drake....@nexr.com>: > Very interesting, BTW. So you try to launch app-master with YARN Container > but your own node-manager without YARN Container, Am I right? > > Drake 민영근 Ph.D > kt NexR > > On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <eastcirc...@postech.ac.kr> > wrote: >> >> Thanks for your input but I need to launch my own node manager >> (different from the Yarn NM) running on each node. >> (which is not explained in the original question) >> >> If I were to launch just a single master with a well-known address, >> ZooKeeper would be a great solution! >> Thanks. >> >> Dongwon Kim >> >> 2015-03-31 10:47 GMT+09:00 Drake민영근 <drake....@nexr.com>: >> > Hi, >> > >> > In these circumstances, there is no easy way to do that. Maybe use >> > workaround. How about using zookeeper for shared storage? The app master >> > create predefined zookeeper node when starting with current machine's IP >> > and >> > Clients always look for that zookeeper node for app master's location. >> > >> > Thanks. >> > >> > >> > Drake 민영근 Ph.D >> > kt NexR >> > >> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim >> > <eastcirc...@postech.ac.kr> >> > wrote: >> >> >> >> Hello, >> >> >> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app >> >> master on a specific node in a YARN cluster in order to open a server >> >> on a predetermined IP address and port. To that end, I wrote a driver >> >> program in which I created a ResourceRequest object and called >> >> setResourceName method to set a hostname, and attached it to a >> >> ApplicationSubmissionContext object by >> >> callingsetAMContainerResourceRequest method. >> >> >> >> I tried several times but couldn't launch the app master on a specific >> >> node. After searching code, I found that RMAppAttemptImpl invalidates >> >> what I've set in ResourceRequest as follows: >> >> >> >> // Currently, following fields are all hard code, >> >> // TODO: change these fields when we want to support >> >> // priority/resource-name/relax-locality specification for AM >> >> containers >> >> // allocation. >> >> appAttempt.amReq.setNumContainers(1); >> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY); >> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY); >> >> appAttempt.amReq.setRelaxLocality(true); >> >> >> >> Is there another way to launch a container for an application master >> >> on a specific node in Hadoop-2.6.0? >> >> >> >> Thanks. >> >> >> >> Dongwon Kim >> > >> > > >
