[
https://issues.apache.org/jira/browse/SPARK-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040409#comment-16040409
]
Sean Owen commented on SPARK-20691:
-----------------------------------
[~maartenkessela...@gmail.com] what I don't want to do is change any behavior
here. This function interprets strings like "700m" as "700 mebibytes" and we
should leave that behavior. Its javadoc looks correct, even though the method
name should probably include "MiB" not "Mb", but I don't think that's worth
changing.
I do think it's worth changing bytesToString. I think it's also worth changing
the javascript in utils.js. This should make most, if not all, the display
consistently and correctly report mebi / gibi / kibibytes, like parsing
routines use.
But here I think we should focus on changing only display.
> Difference between Storage Memory as seen internally and in web UI
> ------------------------------------------------------------------
>
> Key: SPARK-20691
> URL: https://issues.apache.org/jira/browse/SPARK-20691
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.3.0
> Reporter: Jacek Laskowski
> Labels: starter
>
> I set Major priority as it's visible to a user.
> There's a difference in what the size of Storage Memory is managed internally
> and displayed to a user in web UI.
> I found it while answering [How does web UI calculate Storage Memory (in
> Executors tab)?|http://stackoverflow.com/q/43801062/1305344] on StackOverflow.
> In short (quoting the main parts), when you start a Spark app (say
> spark-shell) you see 912.3 MB RAM for Storage Memory:
> {code}
> $ ./bin/spark-shell --conf spark.driver.memory=2g
> ...
> 17/05/07 15:20:50 INFO BlockManagerMasterEndpoint: Registering block manager
> 192.168.1.8:57177 with 912.3 MB RAM, BlockManagerId(driver, 192.168.1.8,
> 57177, None)
> {code}
> but in the web UI you'll see 956.6 MB due to the way the custom JavaScript
> function {{formatBytes}} in
> [utils.js|https://github.com/apache/spark/blob/master/core/src/main/resources/org/apache/spark/ui/static/utils.js#L40-L48]
> calculates the value. That translates to the following Scala code:
> {code}
> def formatBytes(bytes: Double) = {
> val k = 1000
> val i = math.floor(math.log(bytes) / math.log(k))
> val maxMemoryWebUI = bytes / math.pow(k, i)
> f"$maxMemoryWebUI%1.1f"
> }
> scala> println(formatBytes(maxMemory))
> 956.6
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
