1. To remove a state, you need to call "state.remove()". If you return a None in the function, it just means don't output it as the DStream's output, but the state won't be removed if you don't call "state.remove()".
2. For NoSuchElementException, here is the doc for "State.get": /** * Get the state if it exists, otherwise it will throw `java.util.NoSuchElementException`. * Check with `exists()` whether the state exists or not before calling `get()`. * * @throws java.util.NoSuchElementException If the state does not exist. */ On Fri, Jan 29, 2016 at 10:45 AM, Udo Fholl <udofholl1...@gmail.com> wrote: > Hi, > > From the signature of the "mapWithState" method I infer that by returning > a "None.type" (in Scala) the key is removed from the state. Is that so? > Sorry if it is in the docs, but it wasn't entirely clear to me. > > I'm chaining operations and calling "mapWithState" twice (one to > consolidate, then I perform some operations that might, or might not > succeed, and invoke "mapWithState" again). I'm getting this error[1] which > I suppose is because I'm returning "None" in the "mapWithState" function. > > Thank you. > > Best regards, > Udo. > > [1]: java.util.NoSuchElementException: State is not set > at org.apache.spark.streaming.StateImpl.get(State.scala:150) >
