Re: How to recovery DStream from checkpoint directory?

Any kind of changes to the jvm classes will make it fail. By checkpointing the data you mean using checkpoint with updateStateByKey? Here's a similar discussion happened earlier which will clear your doubts i guess

Re: How to recovery DStream from checkpoint directory?

In my understand, here I have only three options to keep the DStream state between redeploys (yes, I'm using updateStateByKey): 1. Use checkpoint. 2. Use my own database. 3. Use both. But none of these options are great: 1. Use checkpoint: I cannot load it after code change. Or I need to keep

Re: How to recovery DStream from checkpoint directory?

: How to recovery DStream from checkpoint directory? In my understand, here I have only three options to keep the DStream state between redeploys (yes, I'm using updateStateByKey): 1. Use checkpoint. 2. Use my own database. 3. Use both. But none of these options are great: 1. Use checkpoint: I

Re: How to recovery DStream from checkpoint directory?

the values preloaded from DB > 2. By cleaning the checkpoint in between upgrades, data is loaded > only once > > Hope this helps, > -adrian > > From: Bin Wang > Date: Thursday, September 17, 2015 at 11:27 AM > To: Akhil Das > Cc: user > Subject: Re: How t

Re: How to recovery DStream from checkpoint directory?

You can't really recover from checkpoint if you alter the code. A better approach would be to use some sort of external storage (like a db or zookeeper etc) to keep the state (the indexes etc) and then when you deploy new code they can be easily recovered. Thanks Best Regards On Wed, Sep 16,

Re: How to recovery DStream from checkpoint directory?

Will StreamingContex.getOrCreate do this work？What kind of code change will make it cannot load？ Akhil Das 于2015年9月16日周三 20:20写道： > You can't really recover from checkpoint if you alter the code. A better > approach would be to use some sort of external storage (like

How to recovery DStream from checkpoint directory?

I'd like to know if there is a way to recovery dstream from checkpoint. Because I stores state in DStream, I'd like the state to be recovered when I restart the application and deploy new code.

Re: How to recovery DStream from checkpoint directory?

And here is another question. If I load the DStream from database every time I start the job, will the data be loaded when the job is failed and auto restart? If so, both the checkpoint data and database data are loaded, won't this a problem? Bin Wang 于2015年9月16日周三 下午8:40写道：