I'm sure this must be a fairly common use-case for spark, yet I have not found a satisfactory discussion of it on the spark website or forum:
I work at a company with a lot of previous-generation server hardware sitting idle-- I want to add this hardware to my spark cluster to increase performance! BUT: It is unclear as to whether the spark master will be able to properly apportion jobs to the slaves if they have differing hardware specs. As I understand, the default spark launch scripts are incompatible with per-node hardware configurations, but it seems I could compose custom spark-conf.sh files for each slave to fully utilize its hardware. Would the master take these per-node configurations into consideration when allocating work? or would the cluster necessarily fall to the lowest-common-hardware-denominator? Is this an area which needs development? I might be willing to look into attempting to introduce this functionality if it is lacking. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/heterogeneous-cluster-hardware-tp11567.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
