Re: Spark on Mesos: Spark issuing hundreds of SUBSCRIBE requests / second and crashing Mesos

2018-07-23 Thread Nimi W
; scheduler client can result in infinite subscribe loop" ( > https://issues.apache.org/jira/browse/MESOS-8171). It can be fixed by > upgrading to a version of libmesos that has the fix. > > Susan > > > On Fri, Jul 13, 2018 at 3:39 PM, Nimi W wrote: > >> I've come acro

Spark on Mesos: Spark issuing hundreds of SUBSCRIBE requests / second and crashing Mesos

2018-07-13 Thread Nimi W
I've come across an issue with Mesos 1.4.1 and Spark 2.2.1. We launch Spark tasks using the MesosClusterDispatcher in cluster mode. On a couple of occasions, we have noticed that when the Spark Driver crashes (to various causes - human error, network error), sometimes, when the Driver is