The memory leak could be related to this <https://issues.apache.org/jira/browse/SPARK-5967> defect that was resolved in Spark 1.2.2 and 1.3.0.
It also was a HashMap causing the issue. -Conor On Wed, Apr 29, 2015 at 12:01 PM, Sean Owen <so...@cloudera.com> wrote: > Please use user@, not dev@ > > This message does not appear to be from your driver. It also doesn't say > you ran out of memory. It says you didn't tell YARN to let it use the > memory you want. Look at the memory overhead param and please search first > for related discussions. > On Apr 29, 2015 11:43 AM, "wyphao.2007" <wyphao.2...@163.com> wrote: > >> Hi, Dear developer, I am using Spark Streaming to read data from kafka, >> the program already run about 120 hours, but today the program failed >> because of driver's OOM as follow: >> >> Container [pid=49133,containerID=container_1429773909253_0050_02_000001] >> is running beyond physical memory limits. Current usage: 2.5 GB of 2.5 GB >> physical memory used; 3.2 GB of 50 GB virtual memory used. Killing >> container. >> >> I set --driver-memory to 2g, In my mind, driver is responsibility for >> job scheduler and job monitor(Please correct me If I'm wrong), Why it using >> so much memory? >> >> So I using jmap to monitor other program(already run about 48 hours): >> sudo /home/q/java7/jdk1.7.0_45/bin/jmap -histo:live 31256, the result as >> follow: >> the java.util.HashMap$Entry and java.lang.Long object using about 600Mb >> memory! >> >> and I also using jmap to monitor other program(already run about 1 hours >> ), the result as follow: >> the java.util.HashMap$Entry and java.lang.Long object doesn't using so >> many memory, But I found, as time goes by, the java.util.HashMap$Entry >> and java.lang.Long object will occupied more and more memory, >> It is driver's memory leak question? or other reason? >> >> Thanks >> Best Regards >> >> >> >> >> >> >> >> >> >>
