Could you just make Hadoop's resource manager (port 8088) available to your users, and they can check available containers that way if they see the launch is stalling?
Another option is to reduce the default # of executors and memory per executor in the launch script to some small fraction of your cluster size, and make it so users can manually ask for more if they need to. It doesn't take a whole lot of workers/memory to build most of your spark code off a sample. Jon On Wed, Feb 15, 2017 at 6:41 AM, Sachin Aggarwal <different.sac...@gmail.com > wrote: > Hi, > > I am trying to create multiple notebooks connecting to spark on yarn. > After starting few jobs my cluster went out of containers. All new notebook > request are in busy state as Jupyter kernel gateway is not getting any > containers for master to be started. > > Some job are not leaving the containers for approx 10-15 mins. so user is > not able to figure out what is wrong, why his kernel is still in busy state > > Is there any property or hack by which I can return valid response to > users that there are no containers left. > > can I label/mark few containers for master equal to max kernel execution I > am allowing in my cluster. so that if new kernel starts he will at least > one container for master. it can be dynamic on priority based. if there is > no container left then yarn can preempt some containers and provide them to > new requests. > > > -- > > Thanks & Regards > > Sachin Aggarwal > 7760502772 >
