Hello, For a while, we're using Spark on Mesos with fine-grained mode in production. Since Spark 2.0 the fine-grained mode is deprecated so we'd shift to dynamic allocation.
When I tried to setup the dynamic allocation I run into the following problem: So I set spark.shuffle.service.enabled = true and spark.dynamicAllocation.enabled = true as the documentation said. We're using Spark on Mesos with spark.executor.uri where we download the pipeline's corresponding Spark version from HDFS. The documentation also says In Mesos coarse-grained mode, run $SPARK_HOME/sbin/start-mesos-shuffle-service.sh on all slave nodes. But how is it possible to launch it before start the application, if the given Spark will be downloaded to the Mesos executor after executor launch but it's looking for the started external shuffle service in advance? Is it possible I can't use spark.executor.uri and spark.dynamicAllocation.enabled together? Thanks in advance! Tamas
