>The memory leak could be related to this <https://issues.apache.org/jira/browse/SPARK-5967> defect that was resolved in Spark 1.2.2 and 1.3.0. @Sean Will it be backported to CDH? I did't find that bug in CDH 5.4 release notes.
2015-04-29 14:51 GMT+02:00 Conor Fennell <conor.fenn...@altocloud.com>: > The memory leak could be related to this > <https://issues.apache.org/jira/browse/SPARK-5967> defect that was > resolved in Spark 1.2.2 and 1.3.0. > > It also was a HashMap causing the issue. > > -Conor > > > > On Wed, Apr 29, 2015 at 12:01 PM, Sean Owen <so...@cloudera.com> wrote: > >> Please use user@, not dev@ >> >> This message does not appear to be from your driver. It also doesn't say >> you ran out of memory. It says you didn't tell YARN to let it use the >> memory you want. Look at the memory overhead param and please search first >> for related discussions. >> On Apr 29, 2015 11:43 AM, "wyphao.2007" <wyphao.2...@163.com> wrote: >> >>> Hi, Dear developer, I am using Spark Streaming to read data from kafka, >>> the program already run about 120 hours, but today the program failed >>> because of driver's OOM as follow: >>> >>> Container [pid=49133,containerID=container_1429773909253_0050_02_000001] >>> is running beyond physical memory limits. Current usage: 2.5 GB of 2.5 GB >>> physical memory used; 3.2 GB of 50 GB virtual memory used. Killing >>> container. >>> >>> I set --driver-memory to 2g, In my mind, driver is responsibility for >>> job scheduler and job monitor(Please correct me If I'm wrong), Why it using >>> so much memory? >>> >>> So I using jmap to monitor other program(already run about 48 hours): >>> sudo /home/q/java7/jdk1.7.0_45/bin/jmap -histo:live 31256, the result as >>> follow: >>> the java.util.HashMap$Entry and java.lang.Long object using about 600Mb >>> memory! >>> >>> and I also using jmap to monitor other program(already run about 1 hours >>> ), the result as follow: >>> the java.util.HashMap$Entry and java.lang.Long object doesn't using so >>> many memory, But I found, as time goes by, the java.util.HashMap$Entry >>> and java.lang.Long object will occupied more and more memory, >>> It is driver's memory leak question? or other reason? >>> >>> Thanks >>> Best Regards >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >
