The lineage information of any computed RDD is persisted to recover incase of failure. In your case, if the input RDD is in the failed node, then SPARK has enough information as to how the input RDD is calculated from file system/scala collection of a transformation from some other RDD. It recomputes and then carries forward with the stage wise computation.
The only checkpointing done by default by SPARK is the lineage of how RDDs are derived…However for RDDs with very long lineage, you can force checkpointing. On Wed, Jan 29, 2014 at 9:16 AM, Ashish Rangole <arang...@gmail.com> wrote: > My understanding is that if the input has been replicated (assuming HDFS) > then the other available copy would be used otherwise or would go back to > the original source. > On Jan 28, 2014 8:34 PM, "nowfats" <nowf...@gmail.com> wrote: > >> Hi, As I know when spark run a job, it will build a DAG. Then split the >> DAG into stages. Each stage has task set, the taskscheduler will send the >> task set to workers and wait until the stage complete then start the next >> stage. If it is true, I want to know when one worker is failed, the >> taskscheduler can resubmit the lineage information and the input RDD to >> another worker. If the input RDD is not on the failed worker node, it is >> worked that new worker node can fetch the RDD from the node which contain >> the input RDD. However, if the input RDD is in failed worker node, how do >> the fault-tolerant function? Is it use checkpoint to store the input RDD to >> file(such as HDFS)? Thanks >> ------------------------------ >> View this message in context: Spark Fault-tolerant >> question<http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Fault-tolerant-question-tp1008.html> >> Sent from the Apache Spark User List mailing list >> archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com. >> > -- *Sai Prasanna. AN* *II M.Tech (CS), SSSIHL* *Entire water in the ocean can never sink a ship, Unless it gets inside.All the pressures of life can never hurt you, Unless you let them in.*
