Re: Caching intermediate results in Spark ML pipeline?

b/master/mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala#L85> > . > > Otherwise, perhaps you can elaborate more on your particular use case for > caching intermediate results and if the current API doesn't support it we > can create a JIRA for it. > > On

Re: Caching intermediate results in Spark ML pipeline?

between multiple runs, you have a few > options: > * cache it first and pass it in as an argument to searchRun > * use a creational pattern like singleton to ensure only one instantiation > > On Tue, Sep 15, 2015 at 12:49 AM, Jingchu Liu <liujing...@gmail.com> > wrote: > &g

Caching intermediate results in Spark ML pipeline?

Hi all, I have a question regarding the ability of ML pipeline to cache intermediate results. I've posted this question on stackoverflow but got no answer, hope someone here can help me out.

Re: Caching intermediate results in Spark ML pipeline?

.setStages API). See associated > JIRAs <https://issues.apache.org/jira/browse/SPARK-4587>. > > Pipeline persistence is on the 1.6 roadmap, JIRA here > <https://issues.apache.org/jira/browse/SPARK-6725>. > > Feynman > > On Mon, Sep 14, 2015 at 9:20 PM, Jingchu Liu