Hi, I am trying to develop an auction based scheduling algorithm for YARN. I have few questions about the same.
Idea is to have an "Auction Manager" inside every "Node Manager" who receive auction requests from "Resource Manager" for every resource request from "Application Master". The cluster nodes are grouped such that each rack has one seller and all other as participants. Based on few parameters/metrics the seller selects a winning participant to serve the resource request. Question 1: -------------- I added a new module for auctions in my local code repository and pretty much covered most of the use cases. But I am stuck at understanding how communication actually takes place between RM and NM or among NMs. I need to include something on lines of send/recv commands. Like how the map/reduce commands are given using some job.xml. Question 2: -------------- All my code additions are in: hadoop-yarn-api; hadoop-yarn-common; hadoop-yarn-server-common; hadoop-yarn-server-nodemanager; hadoop-yarn-server-resourcemanager Would I need other modules like hadoop-yarn-client etc? My guess is I don't think so. Question 3: -------------- Did not really understand the Yarn-Protos. Any place where I can get some decent documentation about this? Any response/help is greatly appreciated. Thanks, Santosh
