[
https://issues.apache.org/jira/browse/SPARK-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173926#comment-14173926
]
Dev Lakhani commented on SPARK-3957:
------------------------------------
Here is my thoughts on a possible approach.
Hi All
The broadcast occurs form the Spark Context to the broadcastmanager and new
Broadcast method. In the first instance, the broadcasted data is stored in the
Block Manager (see HttpBroadCast) of the executor. Any tracking of broadcast
variables must be referenced by the BlockManagerSlaveActor and
BlockManagerMasterActor. In particular UpdateBlockInfo and RemoveBroadcast
should update the total memory in blocks used when blocks are added and removed.
These can then be hooked up to the UI using a new Page like ExecutorsPage and
defining a new methods in the relevant listener such as StorageStatusListener.
These are my initial thoughts for someone new to these components, any other
ideas or approaches?
> Broadcast variable memory usage not reflected in UI
> ---------------------------------------------------
>
> Key: SPARK-3957
> URL: https://issues.apache.org/jira/browse/SPARK-3957
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Web UI
> Affects Versions: 1.0.2, 1.1.0
> Reporter: Shivaram Venkataraman
> Assignee: Nan Zhu
>
> Memory used by broadcast variables are not reflected in the memory usage
> reported in the WebUI. For example, the executors tab shows memory used in
> each executor but this number doesn't include memory used by broadcast
> variables. Similarly the storage tab only shows list of rdds cached and how
> much memory they use.
> We should add a separate column / tab for broadcast variables to make it
> easier to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
