The shuffle service is run out of band from any specific Spark job, and you
only run one on any given node. You need to get the Spark distribution on
each node somehow, then run the shuffle service out of that distribution.
The most common way I see people doing this is via Marathon (using the
Hello,
For a while, we're using Spark on Mesos with fine-grained mode in
production.
Since Spark 2.0 the fine-grained mode is deprecated so we'd shift to
dynamic allocation.
When I tried to setup the dynamic allocation I run into the following
problem:
So I set spark.shuffle.service.enabled =
