[
https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675690#comment-16675690
]
David Vogelbacher commented on SPARK-24437:
-------------------------------------------
[~eyalfa] the cached relations only really take up space on the executors, as
they hold the cached data, whereas the broadcast variable takes up space on the
driver (which eventually OOMs/GCs a lot).
[~mgaido] Yes, I realize that the broadcast variable is used for re-computation
if parts of the cached dataframe get evicted. But couldn't the broadcast
variable also get recomputed in this case? Since we do keep track of the whole
logical plan when caching a dataset, we should always be able to recompute the
broadcast variable when needed.
> Memory leak in UnsafeHashedRelation
> -----------------------------------
>
> Key: SPARK-24437
> URL: https://issues.apache.org/jira/browse/SPARK-24437
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: gagan taneja
> Priority: Critical
> Attachments: Screen Shot 2018-05-30 at 2.05.40 PM.png, Screen Shot
> 2018-05-30 at 2.07.22 PM.png, Screen Shot 2018-11-01 at 10.38.30 AM.png
>
>
> There seems to memory leak withÂ
> org.apache.spark.sql.execution.joins.UnsafeHashedRelation
> We have a long running instance of STS.
> With each query execution requiring Broadcast Join, UnsafeHashedRelation is
> getting added for cleanup in ContextCleaner. This reference of
> UnsafeHashedRelation is being held at some other Collection and not becoming
> eligible for GC and because of this ContextCleaner is not able to clean it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
