Use Structured Streaming. Its aggregation, by definition, is across batches.
On Thu, Feb 27, 2020 at 3:17 PM Something Something <
mailinglist...@gmail.com> wrote:
> We've a Spark Streaming job that calculates some values in each batch.
> What we need to do now is aggregate values across ALL
We've a Spark Streaming job that calculates some values in each batch. What
we need to do now is aggregate values across ALL batches. What is the best
strategy to do this in Spark Streaming. Should we use 'Spark Accumulators'
for this?
