Hello, I'm working with Structured Streaming, and I need a method of keeping a running average based on last 24hours of data. To help with this, I can use Exponential Smoothing, which means I really only need to store 1 value from a previous calculation into the new, and update this variable as calculations carry on.
Implementing this is a much bigger challenge then I ever imagined. I've tried using Accumulators and to Query/Store data to Cassandra after every calculation. Both methods worked somewhat locally , but I don't seem to be able to use these in the Spark Worker Nodes, as I get the error "java.lang.NoClassDefFoundError: Could not initialize class error" both for the accumulator and the cassandra connection libary How can you read/update a variable while doing calculations using Structured Streaming? Thank you
