Yeah, in general if you're changing the jar you can't recover the
checkpoint.
If you're just changing parameters, why not externalize those in a
configuration file so your jar doesn't change? I tend to stick even my
app-specific parameters in an external spark config so everything is in one
place
Good tip. I will try that.
Thank you.
On Wed, Sep 2, 2015 at 6:54 PM, Cody Koeninger wrote:
> Yeah, in general if you're changing the jar you can't recover the
> checkpoint.
>
> If you're just changing parameters, why not externalize those in a
> configuration file so your jar doesn't change?
Hi guys,
I tried to use the configuration file, but it didn't work as I expected. As
part of the Spark Streaming flow, my methods run only when the application
is started the first time. Once I restart the app, it reads from the
checkpoint and all the dstream operations come from the cache. No par
Yeah, it makes sense that parameters that are read only during your
getOrCCreate function wouldn't be re-read, since that function isn't called
if a checkpoint is loaded.
I would have thought changing number of executors and other things used by
spark-submit would work on checkpoint restart. Have
Hi Cody,
Thanks for your answer.
I had already tried to change the spark submit parameters, but I double
checked to reply your answer. Even changing properties file or directly on
the spark-submit arguments, none of them work when the application runs
from the checkpoint. It seems that everything
Yeah, looks like you're right about being unable to change those. Upon
further reading, even though StreamingContext.getOrCreate makes an entirely
new spark conf, Checkpoint will only reload certain properties.
I'm not sure if it'd be safe to include memory / cores among those
properties that get
Thanks Cody.
You confirmed that I'm not doing something wrong. I will keep investigating
and if I find something I let everybody know.
Thanks again.
Regards,
Ricardo
On Mon, Sep 14, 2015 at 6:29 PM, Cody Koeninger wrote:
> Yeah, looks like you're right about being unable to change those. Up