[
https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178068#comment-14178068
]
Patrick Wendell commented on SPARK-4030:
----------------------------------------
Hey Shivaram - IIRC we made this private to protect users from shooting
themselves in the foot and trying to use a broadcast variable that has been
destroy()'d, potentially by another thread or somewhere difficult to track
down. If we expose this it would be good to test it thoroughly (I think we put
some logic in there around this already thinking that we might open it up
later). Also, it might be good to capture the callsite of the destroy call and
echo it back to users if they attempt to use an invalid broadcast variable.
> `destroy` method in Broadcast should be public
> ----------------------------------------------
>
> Key: SPARK-4030
> URL: https://issues.apache.org/jira/browse/SPARK-4030
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
> Affects Versions: 1.1.0, 1.2.0
> Reporter: Shivaram Venkataraman
>
> The destroy method in Broadcast.scala
> [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/broadcast/Broadcast.scala#L91]
> is right now marked as private[spark]
> This prevents long-running applications from cleaning up memory used by
> broadcast variables on the driver. Also as broadcast variables are always
> created with persistence MEMORY_DISK, this slows down jobs when old broadcast
> variables are flushed to disk.
> Making `destroy` public can help applications control the lifetime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
