Hey Jonathan,
Are you referring to disk space used for storing persisted RDD's? For
that, Spark does not bound the amount of data persisted to disk. It's
a similar story to how Spark's shuffle disk output works (and also
Hadoop and other frameworks make this assumption as well for their
shuffle
Nothing so complicated... we are seeing mesos kill off our executors
immediately. When I reroute logging to an NFS directory we have available,
the executors survive fine. As such I am wondering if the spark workers are
getting killed by mesos for exceeding their disk quota (which atm is 0).
This
I'm surprised that I haven't been able to find this via google, but I
haven't...
What is the setting that requests some amount of disk space for the
executors? Maybe I'm misunderstanding how this is configured...
Thanks for any help!
