Hi Rishi, That is exactly why Trigger.Once was created for Structured Streaming. The way we look at streaming is that it doesn't have to be always real time, or 24-7 always on. We see streaming as a workflow that you have to repeat indefinitely. See this blog post for more details! https://databricks.com/blog/2017/05/22/running-streaming-jobs-day-10x-cost-savings.html
Best, Burak On Fri, May 1, 2020 at 2:55 PM Rishi Shah <rishishah.s...@gmail.com> wrote: > Hi All, > > I recently started playing with spark streaming, and checkpoint location > feature looks very promising. I wonder if anyone has an opinion about using > spark streaming with checkpoint location option as a slow batch processing > solution. What would be the pros and cons of utilizing streaming with > checkpoint location feature to achieve fault tolerance in batch processing > application? > > -- > Regards, > > Rishi Shah >
