Re: Avoiding using hostname for YARN nodemanagers

2017-12-05 Thread Alvaro Brandon
created docker network DNS resolution of these hostnames > among the containers is possible. provide --network-alias parameter to add > hostname for DNS > > Check if that works for you. > -Vinay > > > On 5 Dec 2017 9:20 pm, "Alvaro Brandon" <alvarobran...@gmail.co

Avoiding using hostname for YARN nodemanagers

2017-12-05 Thread Alvaro Brandon
Hello: I'm using Docker images to build a YARN cluster. I have a problem when the node managers register with the resource manager. Since they are containers they use the hash that the Docker engine assigns to them as the hostname. *17/12/05 14:56:16 INFO nodemanager.NodeStatusUpdaterImpl:

Choosing a subset of machines to launch Spark Application

2017-02-07 Thread Alvaro Brandon
Hello all: I have the following scenario. - I have a cluster of 50 machines with Hadoop and Spark installed on them. - I want to launch one Spark application through spark submit. However I want this application to run on only a subset of these machines, disregarding data locality. (e.g. 10

Restart number of vcores in YARN

2016-07-15 Thread Alvaro Brandon
Hello everyone: I've changed yarn.nodemanager.resource.cpu-vcores in my yarn-site.xml configuration file and restarted all the yarn and hdfs services. However the nodes doesn't reflect this change in the number of available virtual cores, at least when I query the resource manager API. How can

YARN application start event

2016-07-07 Thread Alvaro Brandon
Hello everyone: I was wondering if there is any way to capture the event of an application starting in YARN. The idea is to implement a Listener that every time a YARN application starts, will query the REST API to get the current memory and cores availables in the cluster. Any ideas on this?