We have a timestamped input stream and we need to share the latest processed timestamp across spark master and slaves. This will be monotonically increasing over time. What is the easiest way to share state across spark machines?
An accumulator is very close to what we need, but since only the driver program can read the accumulator’s value, it won't work. Any suggestions? Thanks, -C
