That will heavily depend on your implementation of state functions on flatMapGroupsWithState, but assuming your state functions respect of the output mode, chaining flatMapGroupsWithState doesn't work - I don't mean it doesn't compile or doesn't run. I mean the query will not produce correct output.
The guide doc of Structured Streaming in Spark 3.0 will contain the limitation of global watermark which explains why the chaining of stateful operators in Structured Streaming doesn't work correctly. Spark 3.0 will also log the warning message if Spark detects the pattern. Reading the new part in doc requires understanding of semantic of watermark and how global watermark is calculated in prior, so a bit advanced topic. (Though I think end users should also need to understand this to make sure their query is "correct".) You can see the change of doc previously in this PR: https://github.com/apache/spark/pull/24890 Thanks, Jungtaek Lim (HeartSaVioR) On Thu, Nov 28, 2019 at 9:55 PM alex770 <aha...@iai.co.il> wrote: > I wrote few examples chaining flatMapGroupsWithState in append mode and it > worked. Other examples did not. > > The question is if the flatMapGroupsWithState chains in append mode are > officially supported. > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
