Hi, folks. I noticed that SSS won't process a waiting batch if there are no batches after that. To put it another way, Spark must always leave one batch on Kafka waiting to be consumed.
There is a JIRA for this at: https://issues.apache.org/jira/browse/SPARK-24156 that says it's resolved in 2.4.0 but my code <https://github.com/PhillHenry/SSSPlayground/blob/Spark2/src/test/scala/uk/co/odinconsultants/sssplayground/windows/TimestampedStreamingSpec.scala> is using 2.4.2 yet I still see Spark reluctant to consume another batch from Kafka if it means there is nothing else waiting to be processed in the topic. Do I have to do something special to exploit the behaviour that SPARK-24156 says it has addressed? Regards, Phillip
