Re: Why do checkpoints work the way they do?

Thanks Tathagata for the clarification. +1 on documenting limitations of checkpoints on structured streaming. Hugo On Mon, Sep 11, 2017 at 7:13 PM, Dmitry Naumenko wrote: > +1 for me for this question. If there any constraints in restoring > checkpoint for Structured

Re: Why do checkpoints work the way they do?

+1 for me for this question. If there any constraints in restoring checkpoint for Structured Streaming, they should be documented. 2017-08-31 9:20 GMT+03:00 张万新 : > So is there any documents demonstrating in what condition can my > application recover from the same

Re: Why do checkpoints work the way they do?

So is there any documents demonstrating in what condition can my application recover from the same checkpoint and in what condition not? Tathagata Das 于2017年8月30日周三 下午1:20写道： > Hello, > > This is an unfortunate design on my part when I was building DStreams :) > >

Re: Why do checkpoints work the way they do?

Hello, This is an unfortunate design on my part when I was building DStreams :) Fortunately, we learnt from our mistakes and built Structured Streaming the correct way. Checkpointing in Structured Streaming stores only the progress information (offsets, etc.), and the user can change their

Why do checkpoints work the way they do?

Hello, I am implementing a spark streaming solution with Kafka and read that checkpoints cannot be used across application code changes - here I tested changes in application code and got the error message as b below -