Hey Udo, mapWithState usually uses much more memory than updateStateByKey since it caches the states in memory.
However, from your description, looks BlockGenerator cannot push data into BlockManager, there may be something wrong in BlockGenerator. Could you share the top 50 objects in the heap dump and the thread dump? On Wed, Feb 3, 2016 at 9:52 AM, Udo Fholl <udofholl1...@gmail.com> wrote: > Hi all, > > I recently migrated from 'updateStateByKey' to 'mapWithState' and now I > see a huge increase of memory. Most of it is a massive "BlockGenerator" > (which points to a massive "ArrayBlockingQueue" that in turns point to a > huge "Object[]"). > > I'm pretty sure it has to do with my code, but I barely changed anything > in the code. Just adapted the function. > > Did anyone run into this? > > Best regards, > Udo. >
