> but Spark on Mesos is certainly lagging behind Spark on YARN regarding the features Spark uses off the scheduler backends -- security, data locality, queues, etc.
If by security you mean Kerberos, we'll be upstreaming that to Apache Spark soon. It's been in DC/OS Spark for a while: https://github.com/mesosphere/spark/commit/73ba2ab8d97510d5475ef9a48c673ce34f7173fa Locality is implemented in a scheduler independent way: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L327, but it is possible that the offer model could result in different placement. I haven't seen any analysis to that effect. YARN queues are very similar to Mesos quota and roles, which Spark supports. We'll also be adding support for revocable resource support sometime soon, which solves the HoL blocking problem, where one Spark app eats up your cluster while others wait. I don't think YARN has a solution for this, but I could be wrong. So, yea, there are some differences, but I think the biggest feature gap right now is really just Kerberos, which will be added soon. There are also other Mesos-specific features we'll be adding soon, such as GPU, CNI, and virtual network but the biggest advantage for running on Mesos is that you can run multi-tenant alongside other Mesos frameworks. On Mon, Jul 25, 2016 at 2:04 PM, Jacek Laskowski <ja...@japila.pl> wrote: > On Mon, Jul 25, 2016 at 10:57 PM, Mich Talebzadeh > <mich.talebza...@gmail.com> wrote: > > > Yarn promises the best resource management I believe. Having said that I > have not used Mesos myself. > > I'm glad you've mentioned it. > > I think Cloudera (and Hortonworks?) guys are doing a great job with > bringing all the features of YARN to Spark and I think Spark on YARN > shines features-wise. > > I'm not in a position to compare YARN vs Mesos for their resource > management, but Spark on Mesos is certainly lagging behind Spark on > YARN regarding the features Spark uses off the scheduler backends -- > security, data locality, queues, etc. (or I might be simply biased > after having spent months with Spark on YARN mostly?). > > Jacek > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > -- Michael Gummelt Software Engineer Mesosphere