Kostas papageorgopoulos created SPARK-11700: -----------------------------------------------
Summary: Possible memory leak at SparkContext jobProgressListener stageIdToData map Key: SPARK-11700 URL: https://issues.apache.org/jira/browse/SPARK-11700 Project: Spark Issue Type: Question Components: Spark Core, SQL Affects Versions: 1.5.1, 1.5.0, 1.5.2 Environment: Ubuntu 14.04 LTS, Oracle JDK 1.8.51 Apache tomcat 8.0.28. Spring 4 Reporter: Kostas papageorgopoulos Priority: Minor I have created a java webapp trying to abstractly Run some Spark Sql jobs that read data from HDFS (join them) and Write them To ElasticSearch using ES hadoop connector. After a Lot of consecutive runs i noticed that my heap space was full so i got an out of heap space error. At the attached file {code} AbstractSparkJobRunner {code} the {code} public final void run(T jobConfiguration, ExecutionLog executionLog) throws Exception {code} runs each time an Spark Sql Job is triggered. So tried to reuse the same SparkContext for a number of consecutive runs. If some rules apply i try to clean up the SparkContext by first calling {code} killSparkAndSqlContext {code}. This code eventually runs {code} synchronized (sparkContextThreadLock) { if (javaSparkContext != null) { LOGGER.info("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! CLEARING SPARK CONTEXT!!!!!!!!!!!!!!!!!!!!!!!!!!!"); javaSparkContext.stop(); javaSparkContext = null; sqlContext = null; System.gc(); } numberOfRunningJobsForSparkContext.getAndSet(0); } {code}. So at some point in time i suppose that if no other SparkSql job should run i should kill the sparkContext and this should be garbage collected from garbage collector. However this is not the case, Even if in my debugger see attached picture {code} SparkContextPossibleMemoryLeakIDEA_DEBUG.png {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org