You can use cache or persist. On Tue, Oct 18, 2016 at 10:11 AM, Yang <teddyyyy...@gmail.com> wrote:
> I'm trying out 2.0, and ran a long job with 10 stages, in spark-shell > > it seems that after all 10 finished successfully, if I run the last, or > the 9th again, > spark reruns all the previous stages from scratch, instead of utilizing > the partial results. > > this is quite serious since I can't experiment while making small changes > to the code. > > any idea what part of the spark framework might have caused this ? > > thanks > Yang > -- Best Regards, Ayan Guha
