[
https://issues.apache.org/jira/browse/SPARK-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040263#comment-16040263
]
Maarten Kesselaers edited comment on SPARK-20691 at 6/7/17 6:24 AM:
--------------------------------------------------------------------
[~srowen], I would like to work on this one.
As far as I can see, the behaviour of Utils.bytesToString should not be
changed since it returns MegaBytes instead of MebiBytes.
Let's first clarify the reason of difference between the shell and Web UI and
then we can discuss how to proceed.
was (Author: mkesselaers):
[~srowen], I would like to work on this one.
As far as I've understood (I'm quite new), the behaviour of
Utils.bytesToString should be changed to return MiB instead of MB and we should
look for all occurences to see if the right value is used?
> Difference between Storage Memory as seen internally and in web UI
> ------------------------------------------------------------------
>
> Key: SPARK-20691
> URL: https://issues.apache.org/jira/browse/SPARK-20691
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.3.0
> Reporter: Jacek Laskowski
> Labels: starter
>
> I set Major priority as it's visible to a user.
> There's a difference in what the size of Storage Memory is managed internally
> and displayed to a user in web UI.
> I found it while answering [How does web UI calculate Storage Memory (in
> Executors tab)?|http://stackoverflow.com/q/43801062/1305344] on StackOverflow.
> In short (quoting the main parts), when you start a Spark app (say
> spark-shell) you see 912.3 MB RAM for Storage Memory:
> {code}
> $ ./bin/spark-shell --conf spark.driver.memory=2g
> ...
> 17/05/07 15:20:50 INFO BlockManagerMasterEndpoint: Registering block manager
> 192.168.1.8:57177 with 912.3 MB RAM, BlockManagerId(driver, 192.168.1.8,
> 57177, None)
> {code}
> but in the web UI you'll see 956.6 MB due to the way the custom JavaScript
> function {{formatBytes}} in
> [utils.js|https://github.com/apache/spark/blob/master/core/src/main/resources/org/apache/spark/ui/static/utils.js#L40-L48]
> calculates the value. That translates to the following Scala code:
> {code}
> def formatBytes(bytes: Double) = {
> val k = 1000
> val i = math.floor(math.log(bytes) / math.log(k))
> val maxMemoryWebUI = bytes / math.pow(k, i)
> f"$maxMemoryWebUI%1.1f"
> }
> scala> println(formatBytes(maxMemory))
> 956.6
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
