How data is distributed while processing in spark cluster?

Hi, I am working on auto scaling spark cluster. I would like to know how master distributes the data to the slaves for processing in detail. Any information on this would be helpful. Thanks, Srujana -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How

Re: How data is distributed while processing in spark cluster?

Thanks for the response. I would also like to know, What happens if a slave node is removed while it is processing some data. Does master send that data for re-processing/resume-process to other slave nodes ? And does it happen with the help of HDFS? Thanks, Srujana -- View this message