Hi, I'm curious how would you do the requirement "by a certain amount of time" without a watermark? How would you know what's current and compute the lag? Let's forget about watermark for a moment and see if it pops up as an inevitable feature :)
"I am trying to filter out records which are lagging behind (based on event time) by a certain amount of time." Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski Mastering Spark SQL https://bit.ly/mastering-spark-sql Spark Structured Streaming https://bit.ly/spark-structured-streaming Mastering Kafka Streams https://bit.ly/mastering-kafka-streams Follow me at https://twitter.com/jaceklaskowski On Fri, Jan 26, 2018 at 7:14 PM, M Singh <mans2si...@yahoo.com.invalid> wrote: > Hi: > > I am trying to filter out records which are lagging behind (based on event > time) by a certain amount of time. > > Is the watermark api applicable to this scenario (ie, filtering lagging > records) or it is only applicable with aggregation ? I could not get a > clear understanding from the documentation which only refers to it's usage > with aggregation. > > Thanks > > Mans >
