Re: Spark Streaming checkpoints and code upgrade

, 2015 at 10:19 AM, Nicolas Monchy <nico...@gumgum.com> > wrote: > >> Hello, >> >> I am using Spark Streaming and the Kafka Direct API and I am >> checkpointing the metadata. >> Checkpoints aren't recoverable if you upgrade code so I am losing the >> las

Spark Streaming checkpoints and code upgrade

Hello, I am using Spark Streaming and the Kafka Direct API and I am checkpointing the metadata. Checkpoints aren't recoverable if you upgrade code so I am losing the last consumed offsets in this case. I know I can build a system to store and load the offsets for each batch but before