Yeah, in general if you're changing the jar you can't recover the checkpoint.
If you're just changing parameters, why not externalize those in a configuration file so your jar doesn't change? I tend to stick even my app-specific parameters in an external spark config so everything is in one place. On Wed, Sep 2, 2015 at 4:48 PM, Ricardo Luis Silva Paiva < ricardo.pa...@corp.globo.com> wrote: > Hi, > > Is there a way to submit an app code change, keeping the checkpoint data > or do I need to erase the checkpoint folder every time I re-submit the > spark app with a new jar? > > I have an app that count pageviews streaming from Kafka, and deliver a > file every hour from the past 24 hours. I'm using reduceByKeyAndWindow with > the reduce and inverse functions set. > > I'm doing some code improvements and would like to keep the data from the > past hours, so when I re-submit a code change, I would keep delivering the > pageviews aggregation without need to wait for 24 hours of new data. > Sometimes I'm just changing the submission parameters, like number of > executors, memory and cores. > > Many thanks, > > Ricardo > > -- > Ricardo Paiva > Big Data / Semântica > *globo.com* <http://www.globo.com> >
