Re: Does mapWithState need checkpointing to be specified in Spark Streaming?
Yes, you do. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Does-mapWithState-need-checkpointing-to-be-specified-in-Spark-Streaming-tp28858p28862.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: Does mapWithState need checkpointing to be specified in Spark Streaming?
OK. Thanks TD. Does stateSnapshots() bring the snapshot of the state of all the keys managed by mapWithState or does it just bring the state of the keys in the current micro batch? Its kind of conflicting because the following link says that it brings the state only for the keys seen in the current batch. But, the code documentation says that it brings the state for all the keys. Also stateSnapshots() does it give the expired key state for the last time if we set it? wordStream.mapWithState(stateSpec).stateSnapshots() https://docs.cloud.databricks.com/docs/spark/1.6/examples/Streaming%20mapWithState.html On Thu, Jul 13, 2017 at 3:00 PM, Tathagata Das <tathagata.das1...@gmail.com> wrote: > Yes. It does. > > On that note, Spark 2.2 (released couple of days ago) adds > mapGroupsWithState in Structured Streaming. That is like mapWithState on > steroids. Just saying. :) > > On Thu, Jul 13, 2017 at 1:01 PM, SRK <swethakasire...@gmail.com> wrote: > >> Hi, >> >> Do we need to specify checkpointing for mapWithState just like we do for >> updateStateByKey? >> >> Thanks, >> Swetha >> >> >> >> -- >> View this message in context: http://apache-spark-user-list. >> 1001560.n3.nabble.com/Does-mapWithState-need-checkpointing- >> to-be-specified-in-Spark-Streaming-tp28858.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> >
Re: Does mapWithState need checkpointing to be specified in Spark Streaming?
Yes. It does. On that note, Spark 2.2 (released couple of days ago) adds mapGroupsWithState in Structured Streaming. That is like mapWithState on steroids. Just saying. :) On Thu, Jul 13, 2017 at 1:01 PM, SRK <swethakasire...@gmail.com> wrote: > Hi, > > Do we need to specify checkpointing for mapWithState just like we do for > updateStateByKey? > > Thanks, > Swetha > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/Does-mapWithState-need- > checkpointing-to-be-specified-in-Spark-Streaming-tp28858.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
Does mapWithState need checkpointing to be specified in Spark Streaming?
Hi, Do we need to specify checkpointing for mapWithState just like we do for updateStateByKey? Thanks, Swetha -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Does-mapWithState-need-checkpointing-to-be-specified-in-Spark-Streaming-tp28858.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org