I’d suggest something like Apache YARN, or Apache Mesos with Marathon or
something similar to allow for management, in particular restart on failure.
mn
On Aug 13, 2014, at 7:15 PM, Tobias Pfeiffer t...@preferred.jp wrote:
Hi,
On Thu, Aug 14, 2014 at 5:49 AM, salemi alireza.sal...@udo.edu
You also need to ensure you're using checkpointing and support recreating the
context on driver failure as described in the docs here:
http://spark.apache.org/docs/latest/streaming-programming-guide.html#failure-of-the-driver-node
From: Matt Narrell
Hi All,
what is the best way to make a spark streaming driver highly available. I
would like the backup driver to pickup the processing if the primary driver
dies.
Thanks,
Ali
--
View this message in context:
Hi,
On Thu, Aug 14, 2014 at 5:49 AM, salemi alireza.sal...@udo.edu wrote:
what is the best way to make a spark streaming driver highly available.
I would also be interested in that. In particular for Streaming
applications where the Spark driver is running for a long time, this might
be
