Structured Streaming internally maintains one global watermark by taking a min of the two watermarks. Thats why one gets reported. In Spark 2.4, there will be the option of choosing max instead of min.
Just curious. Why do you have to two watermarks? Whats the query like. TD On Thu, Aug 9, 2018 at 3:15 PM, subramgr <subramanian.gir...@gmail.com> wrote: > Hi, > > We have two *flatMapGroupWithState* in our job and we have two > *withWatermark* > > We are getting the event max time, event time and watermarks from > *QueryProgressEvent*. > > Right now it just returns one *watermark* value. > > Are two watermarks maintained by Spark or just one. > If one which one > If one watermark is maintained per *Dataframe* how do I get the values for > them ? > > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
