By the way, is there any plan to make a pluggable backend for checkpointing? We might be interested in writing a, for example, Cassandra backend.
On Sat, Jan 25, 2014 at 9:49 PM, Xia, Junluan <junluan....@intel.com> wrote: > Hi all > > The description about this Bug submitted by Matei is as following > > > The tipping point seems to be around 50. We should fix this by checkpointing > the RDDs every 10-20 iterations to break the lineage chain, but checkpointing > currently requires HDFS installed, which not all users will have. > > We might also be able to fix DAGScheduler to not be recursive. > > > regards, > Andrew > -- -- Evan Chan Staff Engineer e...@ooyala.com |
