Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/15949#discussion_r89237754 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala --- @@ -235,4 +236,49 @@ class StreamingAggregationSuite extends StreamTest with BeforeAndAfterAll { CheckLastBatch(("a", 30), ("b", 3), ("c", 1)) ) } + + test("prune results by time, complete mode") { + import testImplicits._ + import StreamingAggregationSuite._ + clock = new StreamManualClock + + val inputData = MemoryStream[Long] + + val aggregated = + inputData.toDF() + .groupBy($"value") + .agg(count("*")) + .as[(Long, Long)] + .where('value >= current_timestamp().cast("long") - 10L) + + testStream(aggregated, Complete)( + StartStream(ProcessingTime("10 seconds"), triggerClock = clock), + + // advance clock to 10 seconds + AddData(inputData, 0L, 5L, 5L, 10L), + AdvanceManualClock(10 * 1000), + AssertOnQuery { _ => clock.getTimeMillis() === 10 * 1000 }, + CheckLastBatch((0L, 1), (5L, 2), (10L, 1)), + + // advance clock to 20 seconds + AddData(inputData, 15L, 15L, 20L), + AdvanceManualClock(10 * 1000), + CheckLastBatch((10L, 1), (15L, 2), (20L, 1)), + + // advance clock to 30 seconds + AddData(inputData, 0L), + AdvanceManualClock(10 * 1000), + CheckLastBatch((20L, 1)), + + // advance clock to 40 seconds + AddData(inputData, 25L, 30L, 40L, 45L), + AdvanceManualClock(10 * 1000), + CheckLastBatch((30L, 1), (40L, 1), (45L, 1)) --- End diff -- I dont see how this tests whether the watermark, and timestamp is recovered correctly from log. nothing is being stopped. I think there should be another test that actually stops the stream, drops the complete mode table and restarts it from checkpoint, and see if the table gets recreated to the same value. something like ``` val query = inputData.toDF().... queryName("aggregates").option("checkpointLocation", cpPath).start() memoryStream.add(....) query.processAllAvailable() checkAnswer(table("aggregates"), expectedResults) query.stop() spark.drop("aggregates") clock.add(1000000) val query2 = inputData.toDF().... queryName("aggregates").option("checkpointLocation", cpPath).start() query.processAllAvailable() checkAnswer(table("aggregates"), expectedResults) // verify that increasing the clock while query was down does not change result of the last batch ``` See this for more inspiration - https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala#L472
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org