Excellent. Is there an example of this somewhere?
Sent from my iPhone
On Nov 6, 2014, at 1:43 AM, Sean Owen so...@cloudera.com wrote:
Broadcast vars should work fine in Spark streaming. Broadcast vars are
immutable however. If you have some info to cache which might change
from batch to
My use case has one large data stream (DS1) that obviously maps to a DStream.
The processing of DS1 involves filtering it for any of a set of known
values, which will change over time, though slowly by streaming standards.
If the filter data were static, it seems to obviously map to a broadcast
Broadcast vars should work fine in Spark streaming. Broadcast vars are
immutable however. If you have some info to cache which might change
from batch to batch, you should be able to load it at the start of
your 'foreachRDD' method or equivalent. That's simple and works
assuming your batch
