Re: Spark Streaming checkpoints and code upgrade

Its pretty much impossible to do across arbitrary code changes. For that, the best way is to go forward is the store and load the offsets yourselves. On Wed, Sep 9, 2015 at 10:19 AM, Nicolas Monchy wrote: > Hello, > > I am using Spark Streaming and the Kafka Direct API and I

Re: Spark Streaming checkpoints and code upgrade

Ok. Thank you for the reply. On Wed, Sep 9, 2015 at 11:40 AM, Tathagata Das wrote: > Its pretty much impossible to do across arbitrary code changes. For that, > the best way is to go forward is the store and load the offsets yourselves. > > On Wed, Sep 9, 2015 at 10:19 AM,

Spark Streaming checkpoints and code upgrade

Hello, I am using Spark Streaming and the Kafka Direct API and I am checkpointing the metadata. Checkpoints aren't recoverable if you upgrade code so I am losing the last consumed offsets in this case. I know I can build a system to store and load the offsets for each batch but before