Spark off heap memory leak on Yarn with Kafka direct stream

Hi, I am running spark streaming 1.4.0 on Yarn (Apache distribution 2.6.0) with java 1.8.0_45 and also Kafka direct stream. I am also using spark with scala 2.11 support. The issue I am seeing is that both driver and executor containers are gradually increasing the physical memory usage till

Re: Spark off heap memory leak on Yarn with Kafka direct stream

:43 am, Cody Koeninger c...@koeninger.org wrote: Does the issue only happen when you have no traffic on the topic? Have you profiled to see what's using heap space? On Mon, Jul 13, 2015 at 1:05 PM, Apoorva Sareen apoorva.sar...@gmail.com mailto:apoorva.sar...@gmail.com wrote: Hi, I am