Re: [Spark Streaming] How do we reset the updateStateByKey values.

of updates is an resetState message. If now, continue summing the others. I can provide scala samples, my java is beyond rusty :) -adrian From: Uthayan Suthakar Date: Friday, October 23, 2015 at 2:10 PM To: Sander van Dijk Cc: user Subject: Re: [Spark Streaming] How do we reset

Re: [Spark Streaming] How do we reset the updateStateByKey values.

etState message. If now, continue summing the > others. > > I can provide scala samples, my java is beyond rusty :) > > -adrian > > From: Uthayan Suthakar > Date: Friday, October 23, 2015 at 2:10 PM > To: Sander van Dijk > Cc: user > Subject: Re: [Spark Strea

Re: [Spark Streaming] How do we reset the updateStateByKey values.

Hi Sander, Thank you for your very informative email. From your email, I've learned a quite a bit. >>>Is the condition determined somehow from the data coming through streamLogs, and is newData streamLogs again (rather than a whole data source?) No, they are two different Streams. I have two

Re: [Spark Streaming] How do we reset the updateStateByKey values.

I don't think it is possible in the way you try to do it. It is important to remember that the statements you mention only set up the stream stages, before the stream is actually running. Once it's running, you cannot change, remove or add stages. I am not sure how you determine your condition

Re: [Spark Streaming] How do we reset the updateStateByKey values.

I need to take the value from a RDD and update the state of the other RDD. Is this possible? On 22 October 2015 at 16:06, Uthayan Suthakar wrote: > Hello guys, > > I have a stream job that will carryout computations and update the state > (SUM the value). At some

[Spark Streaming] How do we reset the updateStateByKey values.

Hello guys, I have a stream job that will carryout computations and update the state (SUM the value). At some point, I would like to reset the state. I could drop the state by setting 'None' but I don't want to drop it. I would like to keep the state but update the state. For example: