If I understand correctly, Trigger.once executes only one micro-batch and terminates, that's all. Your understanding of structured streaming applies there as well.
It's like a hybrid approach as bringing incremental processing from micro-batch but having processing interval as batch. That said, while it enables to get both sides of benefits, it's basically structured streaming, inheriting all the limitations on the structured streaming, compared to the batch query. Spark 3.0.0 will bring some change on Trigger.once (SPARK-30669 [1]) - Trigger.once will "ignore" the read limit per micro-batch on data source (like maxOffsetsPerTrigger) and process all available input as possible. (Data sources should migrate to the new API to take effect, but works for built-in data sources like file and Kafka.) 1. https://issues.apache.org/jira/browse/SPARK-30669 2020년 5월 2일 (토) 오후 5:35, Magnus Nilsson <ma...@kth.se>님이 작성: > I've always had a question about Trigger.Once that I never got around to > ask or test for myself. If you have a 24/7 stream to a Kafka topic. > > Will Trigger.Once get the last offset(s) when it starts and then quit once > it hits this offset(s) or will the job run until no new messages is added > to the topic for a particular amount of time? > > br, > > Magnus > > On Sat, May 2, 2020 at 1:22 AM Burak Yavuz <brk...@gmail.com> wrote: > >> Hi Rishi, >> >> That is exactly why Trigger.Once was created for Structured Streaming. >> The way we look at streaming is that it doesn't have to be always real >> time, or 24-7 always on. We see streaming as a workflow that you have to >> repeat indefinitely. See this blog post for more details! >> >> https://databricks.com/blog/2017/05/22/running-streaming-jobs-day-10x-cost-savings.html >> >> Best, >> Burak >> >> On Fri, May 1, 2020 at 2:55 PM Rishi Shah <rishishah.s...@gmail.com> >> wrote: >> >>> Hi All, >>> >>> I recently started playing with spark streaming, and checkpoint location >>> feature looks very promising. I wonder if anyone has an opinion about using >>> spark streaming with checkpoint location option as a slow batch processing >>> solution. What would be the pros and cons of utilizing streaming with >>> checkpoint location feature to achieve fault tolerance in batch processing >>> application? >>> >>> -- >>> Regards, >>> >>> Rishi Shah >>> >>
