You use a windowed aggregation for this On Tue, May 15, 2018, 09:23 Martin Engen <martin.en...@outlook.com> wrote:
> Hello, > > > > I'm working with Structured Streaming, and I need a method of keeping a > running average based on last 24hours of data. > > To help with this, I can use Exponential Smoothing, which means I really > only need to store 1 value from a previous calculation into the new, and > update this variable as calculations carry on. > > > > Implementing this is a much bigger challenge then I ever imagined. > > > > > > I've tried using Accumulators and to Query/Store data to Cassandra after > every calculation. Both methods worked somewhat locally , but I don't seem > to be able to use these in the Spark Worker Nodes, as I get the error > "java.lang.NoClassDefFoundError: Could not initialize class error" both > for the accumulator and the cassandra connection libary > > > > How can you read/update a variable while doing calculations using > Structured Streaming? > > > Thank you > > > >
