[
https://issues.apache.org/jira/browse/SPARK-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14009293#comment-14009293
]
Aaron Davidson commented on SPARK-1855:
---------------------------------------
I agree that significant improvements can be made to Spark's block replication
model, but there's no reason it shouldn't "work" (albeit with potentially poor
write performance and fewer guarantees than one would like) if you increase the
replication level higher than 2, which is possible using
[StorageLevel#apply|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/StorageLevel.scala#L155].
> Provide memory-and-local-disk RDD checkpointing
> -----------------------------------------------
>
> Key: SPARK-1855
> URL: https://issues.apache.org/jira/browse/SPARK-1855
> Project: Spark
> Issue Type: New Feature
> Components: MLlib, Spark Core
> Affects Versions: 1.0.0
> Reporter: Xiangrui Meng
>
> Checkpointing is used to cut long lineage while maintaining fault tolerance.
> The current implementation is HDFS-based. Using the BlockRDD we can create
> in-memory-and-local-disk (with replication) checkpoints that are not as
> reliable as HDFS-based solution but faster.
> It can help applications that require many iterations.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
