UpdateStateByKey will run the update function on every interval, even if the incoming batch is empty. Is there a way to prevent that? If the incoming DStream contains no RDDs (or RDDs of count 0) then I don't want my update function to run.
Note that this is different from running the update function and writing an if (!newValues.isEmpty) inside the function. Thanks, Michael ----- Michael Vogiatzis @mvogiatzis -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Best-way-to-avoid-updateStateByKey-from-running-without-data-tp23762.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org