Well if the polling is application side, then each application running on the
cluster will have to check for cluster pressure rather then one check that
triggers an event to all applications running on the cluster - which is less
than ideal. Do you know of a way I can communicate between the ResourceManager
and the Client API? or can you think of another way I can do this?
I am working on a project to add a "cooperative preemption" feature to Spark
framework where a Spark application can release executors that it is using when
there is resource pressure on the cluster. Currently the only way a spark app
can release executors is if an executor has been idle for a timeout period.
Basically a spark app will have a fair share value along with its maximum and
minimum number of executors. If the spark app has more than its fair share when
it receives a cluster pressure notification - it can release its "extra"
executors so that they can be used by other applications on the cluster (hence
the cooperative aspect).
The use case for this being interactive jobs. Rather than allocate a fixed
amount of resources for say a batch job and interactive jobs, why not just
dynamically shrink the batch job whenever you need to run interactive jobs.
Hopefully that gives you some insight as to what I am trying to do.
Regards,
Charles.
> From: garlanaganarasi...@huawei.com
> To: yarn-dev@hadoop.apache.org
> Subject: RE: Sending an event from YARN to applications running on the cluster
> Date: Thu, 14 Jan 2016 13:24:14 +0000
>
> Hi Charles,
>
> Ok but should it not be the reverse way : polling the other way round to get
> these stats from the REST (preferable) /RPC and do this calculation in the
> application or the manager side ?
> May be if there is any short fall in information from REST for an app then we
> need to expose them. Already app stats provides allocated Mem & vcores and
> also the apps asks. through this i think we can do to an extent some
> calculations
>
> Also i was curios to know what exactly can application like spark do if the
> app's resource pressure is greater than a particular threshold ?
>
> Regards,
> + Naga
> ________________________________________
> From: Charlie Wright [charliewri...@live.ca]
> Sent: Thursday, January 14, 2016 08:55
> To: yarn-dev@hadoop.apache.org
> Subject: RE: Sending an event from YARN to applications running on the cluster
>
> Well im trying to send my own event, a ResourcePressureEvent. I have a thread
> in the ResourceManager that I have created that periodically checks for
> "resource pressure" - which is whenever the (pending resources + used
> resources / total resources >= threshold) - and whenever there is "resource
> pressure" I want to send an event to applications running on the cluster
> (specifically Spark applications).
>
> Charles.
>
> > From: garlanaganarasi...@huawei.com
> > To: yarn-dev@hadoop.apache.org
> > Subject: RE: Sending an event from YARN to applications running on the
> > cluster
> > Date: Thu, 14 Jan 2016 03:15:06 +0000
> >
> > Hi Charles,
> >
> > What YARN event needs to be sent ? may be some info on it will be useful to
> > help you.
> >
> > Regards,
> > + Naga
> > ________________________________________
> > From: Charlie Wright [charliewri...@live.ca]
> > Sent: Thursday, January 14, 2016 08:01
> > To: yarn-dev@hadoop.apache.org
> > Subject: Sending an event from YARN to applications running on the cluster
> >
> > Hello,
> >
> > I was wondering how I could send an event from YARN to Spark applications
> > running on the cluster. From what I gather, Spark uses the AMRMClient class
> > to interface with YARN. How would I go about sending an event from the
> > ResourceManager to the AMRMClient?
> >
> > Any help is appreciated.
> >
> > Thanks,
> > Charles.
