Hi All, A cluster of one spark driver and multiple executors(5) is setup with redis for spark processed data storage and s3 is used for checkpointing. I have a couple of queries about this setup.
1) How to analyze what part of code executes on Spark Driver and what part of code executes on the executors? 2) As Spark driver gets results from spark executors, should Spark executors have any access to redis storage? ( My guess is yes, because executors may need data from redis for further calculations) 3) Should the spark executors have access to the checkpoint storage s3? 4) Can anyone share their checkpoint recovery strategy for s3? Thanks, Puneet -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
