Hi, I'm working on a Spark Streaming application and I would like to know what is the best storage to use for checkpointing.
For testing purposes we're are using NFS between the worker, the master and the driver program (in client mode), but we have some issues with the CheckpointWriter (1 thread dedicated). *My understanding is that NFS is not a good candidate for this usage.* 1. What is the best solution for checkpointing and what are the alternatives ? 2. Does checkpointings directories need to be shared by the driver application and the workers too ? Thanks for your replies -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Checkpointing-solutions-tp23932.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
