Shuffle is an auxiliary service in the NM. You might want to take a look at AuxServices.java in the NodeManager codebase.
Also, you can add config parameters to <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce.shuffle</value> </property> (config knob in NodeManagers) to add more configurable services in the NodeManager. Hope that helps. mahadev On Oct 11, 2011, at 2:05 PM, <[email protected]> <[email protected]> wrote: > Hi, > > As part of MPI implementation in Yarn (aka Hamster), I was looking at > refactoring some of the functionality into node manager services, so that > it can be reused by other frameworks too. (Based on the discussion with > some folks led me to believe that suffle etc is also being planned as NM > service, separating it from deep integration with node manager.) > > I could not find the documentation for building NM services, in the MR-279 > architecture PDF. Does the service need to reside inside the NM jvm ? How > does NM talk to the service ? How do containers discover the service ? (I > know I can look at the code, but if there is already some docs (or even > philosophical musings :-), it will make my life easier. > > Thanks, > > - milind > > --- > Milind Bhandarkar > Greenplum Labs, EMC > (Disclaimer: Opinions expressed in this email are those of the author, and > do not necessarily represent the views of any organization, past or > present, the author might be affiliated with.) > > >
