I am doing the following aggregation on the data val channelChangesAgg = tunerDataJsonDF .withWatermark("ts2", "10 seconds") .groupBy(window(col("ts2"),"10 seconds"), col("env"), col("servicegroupid")) .agg(count("linetransactionid") as "count1")
The only constraint here is that the data is backdated; even though the data is chronologically ordered, the ts2 will be a old date. Given this condition, will the watermarking and aggregation still work ? -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org