Hi Antony, Unfortunately, all executors for any single Spark application must have the same amount of memory. It's possibly to configure YARN with different amounts of memory for each host (using yarn.nodemanager.resource.memory-mb), so other apps might be able to take advantage of the extra memory.
-Sandy On Mon, Jan 26, 2015 at 8:34 AM, Michael Segel <msegel_had...@hotmail.com> wrote: > If you’re running YARN, then you should be able to mix and max where YARN > is managing the resources available on the node. > > Having said that… it depends on which version of Hadoop/YARN. > > If you’re running Hortonworks and Ambari, then setting up multiple > profiles may not be straight forward. (I haven’t seen the latest version of > Ambari) > > So in theory, one profile would be for your smaller 36GB of ram, then one > profile for your 128GB sized machines. > Then as your request resources for your spark job, it should schedule the > jobs based on the cluster’s available resources. > (At least in theory. I haven’t tried this so YMMV) > > HTH > > -Mike > > On Jan 26, 2015, at 4:25 PM, Antony Mayi <antonym...@yahoo.com.INVALID> > wrote: > > should have said I am running as yarn-client. all I can see is specifying > the generic executor memory that is then to be used in all containers. > > > On Monday, 26 January 2015, 16:48, Charles Feduke < > charles.fed...@gmail.com> wrote: > > > > You should look at using Mesos. This should abstract away the individual > hosts into a pool of resources and make the different physical > specifications manageable. > > I haven't tried configuring Spark Standalone mode to have different specs > on different machines but based on spark-env.sh.template: > > # - SPARK_WORKER_CORES, to set the number of cores to use on this machine > # - SPARK_WORKER_MEMORY, to set how much total memory workers have to give > executors (e.g. 1000m, 2g) > # - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. > "-Dx=y") > it looks like you should be able to mix. (Its not clear to me whether > SPARK_WORKER_MEMORY is uniform across the cluster or for the machine where > the config file resides.) > > On Mon Jan 26 2015 at 8:07:51 AM Antony Mayi <antonym...@yahoo.com.invalid> > wrote: > > Hi, > > is it possible to mix hosts with (significantly) different specs within a > cluster (without wasting the extra resources)? for example having 10 nodes > with 36GB RAM/10CPUs now trying to add 3 hosts with 128GB/10CPUs - is there > a way to utilize the extra memory by spark executors (as my understanding > is all spark executors must have same memory). > > thanks, > Antony. > > > > >
