I am doing the following aggregation on the data
val channelChangesAgg = tunerDataJsonDF
.withWatermark("ts2", "10 seconds")
.groupBy(window(col("ts2"),"10 seconds"),
col("env"),
col("servicegroupid"))
.agg(count("linetransactionid") as "count1")
The only constraint here is that the data is backdated; even though the data
is chronologically ordered, the ts2 will be a old date. Given this
condition, will the watermarking and aggregation still work ?
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
