[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671686#comment-16671686 ]
David Vogelbacher commented on SPARK-24437: ------------------------------------------- Hey [~mgaido], I am seeing something similar in one of our long running applications. The longer it runs the higher heap usage in the driver grows. Taking a heap dump, most of its space is taken up by {{UnsafeHashedRelation}} instances. Looking at their paths to GC, they are strongly referenced by {{InMemoryRelation}} instances: !Screen Shot 2018-11-01 at 10.38.30 AM.png! So it looks like if dataframes are cached the {{UnsafeHashedRelation}}s are never cleaned up, because they are strongly references by generated codegen plan? The context cleaner can only clean them up when they are no longer strongly referenced. > Memory leak in UnsafeHashedRelation > ----------------------------------- > > Key: SPARK-24437 > URL: https://issues.apache.org/jira/browse/SPARK-24437 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.0 > Reporter: gagan taneja > Priority: Critical > Attachments: Screen Shot 2018-05-30 at 2.05.40 PM.png, Screen Shot > 2018-05-30 at 2.07.22 PM.png, Screen Shot 2018-11-01 at 10.38.30 AM.png > > > There seems to memory leak with > org.apache.spark.sql.execution.joins.UnsafeHashedRelation > We have a long running instance of STS. > With each query execution requiring Broadcast Join, UnsafeHashedRelation is > getting added for cleanup in ContextCleaner. This reference of > UnsafeHashedRelation is being held at some other Collection and not becoming > eligible for GC and because of this ContextCleaner is not able to clean it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org