Hi Susan, yes, agree with you regarding resource accounting. Imho, in this case shuffle service must run on node no matter what resources are available(same as we don't account for resources that "system" takes - mesos agent, OS itself and any other process that is running on same machine)
One additional argument against managing it with puppet/chef is that this management becomes "leaked abstraction": usually we submit spark frameworks through mesos and give it any spark distribution uri, while to get this shuffle service running as daemon on every node I need to install specific version of spark distribution on this node and then when upgrading spark version it's not enough to give new uri to mesos, I need to create new shuffle service which uses new spark distro(and then port/dir/other conflicts should be resolved) -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
