I have been able to run spark submission in docker container (HOST network) through Marathon on mesos and target to Mesos cluster (zk address) for at least Spark 1.6, 1.5.2 over Mesos 0.26, 0.27.
I do need to define SPARK_PUBLIC_DNS and SPARK_LOCAL_IP so that the spark driver can announce the right IP address. However, on Mesos 0.28, the spark framework will fail with "Failed to shutdown socket with fd 54: Transport endpoint is not connected <https://www.google.com/search?client=safari&rls=en&q=Failed+to+shutdown+socket+with+fd+54:+Transport+endpoint+is+not+connected&ie=UTF-8&oe=UTF-8>”. Eventually, I got the problem bypassed by defining additional LIBPROCESS_IP Please let me know if the behavior is as expected. If it is, it will be good to document the requirement on the Spark Mesos cluster website. Thank you. Yang.
