Re: Is it required to remove checkpoint when submitting a code change?

Thanks Cody. You confirmed that I'm not doing something wrong. I will keep investigating and if I find something I let everybody know. Thanks again. Regards, Ricardo On Mon, Sep 14, 2015 at 6:29 PM, Cody Koeninger wrote: > Yeah, looks like you're right about being unable

Re: Is it required to remove checkpoint when submitting a code change?

Yeah, looks like you're right about being unable to change those. Upon further reading, even though StreamingContext.getOrCreate makes an entirely new spark conf, Checkpoint will only reload certain properties. I'm not sure if it'd be safe to include memory / cores among those properties that

Re: Is it required to remove checkpoint when submitting a code change?

Hi Cody, Thanks for your answer. I had already tried to change the spark submit parameters, but I double checked to reply your answer. Even changing properties file or directly on the spark-submit arguments, none of them work when the application runs from the checkpoint. It seems that

Re: Is it required to remove checkpoint when submitting a code change?

Yeah, it makes sense that parameters that are read only during your getOrCCreate function wouldn't be re-read, since that function isn't called if a checkpoint is loaded. I would have thought changing number of executors and other things used by spark-submit would work on checkpoint restart.

Re: Is it required to remove checkpoint when submitting a code change?

Hi guys, I tried to use the configuration file, but it didn't work as I expected. As part of the Spark Streaming flow, my methods run only when the application is started the first time. Once I restart the app, it reads from the checkpoint and all the dstream operations come from the cache. No

Re: Is it required to remove checkpoint when submitting a code change?

Good tip. I will try that. Thank you. On Wed, Sep 2, 2015 at 6:54 PM, Cody Koeninger wrote: > Yeah, in general if you're changing the jar you can't recover the > checkpoint. > > If you're just changing parameters, why not externalize those in a > configuration file so your

Is it required to remove checkpoint when submitting a code change?

Hi, Is there a way to submit an app code change, keeping the checkpoint data or do I need to erase the checkpoint folder every time I re-submit the spark app with a new jar? I have an app that count pageviews streaming from Kafka, and deliver a file every hour from the past 24 hours. I'm using

Re: Is it required to remove checkpoint when submitting a code change?

Yeah, in general if you're changing the jar you can't recover the checkpoint. If you're just changing parameters, why not externalize those in a configuration file so your jar doesn't change? I tend to stick even my app-specific parameters in an external spark config so everything is in one