[ https://issues.apache.org/jira/browse/SPARK-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292314#comment-16292314 ]
zuotingbing commented on SPARK-22793: ------------------------------------- yes the master branch also has this problem,but the different is so big between branch master to 2.0 . i am not sure this can be merged to the master branch. > Memory leak in Spark Thrift Server > ---------------------------------- > > Key: SPARK-22793 > URL: https://issues.apache.org/jira/browse/SPARK-22793 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.2 > Reporter: zuotingbing > Priority: Critical > > 1. Start HiveThriftServer2. > 2. Connect to thriftserver through beeline. > 3. Close the beeline. > 4. repeat step2 and step 3 for several times, which caused the leak of Memory. > we found there are many directories never be dropped under the path > {code:java} > hive.exec.local.scratchdir > {code} and > {code:java} > hive.exec.scratchdir > {code} , as we know the scratchdir has been added to deleteOnExit when it be > created. So it means that the cache size of FileSystem deleteOnExit will keep > increasing until JVM terminated. > In addition, we use > {code:java} > jmap -histo:live [PID] > {code} to printout the size of objects in HiveThriftServer2 Process, we can > find the object "org.apache.spark.sql.hive.client.HiveClientImpl" and > "org.apache.hadoop.hive.ql.session.SessionState" keep increasing even though > we closed all the beeline connections, which caused the leak of Memory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org