The 2 operations can't be used inside one another. If you need something like an all time average then you need to keep a tuple (sum, count) to which you add all the new values that come in every batch. The average is then just a map on the state DStream.
Makes sense? have I guessed your use case? Sent from my iPhone > On 24 Sep 2015, at 19:47, swetha <swethakasire...@gmail.com> wrote: > > Hi, > > How to use reduceByKey inside updateStateByKey? Suppose I have a bunch of > keys for which I need to do sum and average inside the updateStateByKey by > joining with old state. How do I accomplish that? > > > Thanks, > Swetha > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/reduceByKey-inside-updateStateByKey-in-Spark-Streaming-tp24808.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
