1.4.0 resolves the problem.
The total classes loaded for an updateStateByKey over Int and String types
does not increase.
The total classes loaded for an updateStateByKey over case classes does
increase over time, but
the processing remains stable. Both memory consumption and CPU load remain
Further experimentation indicates these problems only occur when master is
local[*].
There are no issues if a standalone cluster is used.
--
View this message in context:
Apologies for the repeat. The first was rejected by the submission
process
I created a simple Spark streaming program using updateStateByKey.
The domain is represented by case classes for clarity, type safety, etc.
Spark job continuously loads new classes, which are removed by GC to
maintain
a
