RE: Spark 2.0 Continuous Processing

2016-06-30 Thread kmat
In a continuous processing pipeline with dataframes is there any way to checkpoint the processing state (by the user) at periodic intervals. The thought process behind this is to rewind to any particular checkpoint and then fast forward processing thereon. Date: Wed, 29 Jun 2016 23:17:47 -0700

Spark 2.0 Continuous Processing

2016-06-24 Thread kmat
Is there a way to checkpoint sink(s) to facilitate rewind processing from a specific offset. For example a continuous query aggregated by month. On the 10 month would like to re-compute information between 4th to 8th months. -- View this message in context: