[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18229 gentle ping @mkesselaers --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18229 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18229 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18229 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18229 gentle ping @mkesselaers --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/18229 @mkesselaers are you still working on this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/18229 I agree with changing the base 1000 to 1024. About `MiB` and `MB`, strictly speaking, they are different. But I think in the computing technology world, most people say Megabyte (MB) instead of Mebibytes even though they know the difference. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18229 @kishorvpatil @wzhfy do you have any thoughts on this? it touches some code you created --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18229 **[Test build #3784 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3784/testReport)** for PR 18229 at commit [`dd9cc4d`](https://github.com/apache/spark/commit/dd9cc4def988053b670bc253e2ee5b0f53940de3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18229 **[Test build #3784 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3784/testReport)** for PR 18229 at commit [`dd9cc4d`](https://github.com/apache/spark/commit/dd9cc4def988053b670bc253e2ee5b0f53940de3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user mkesselaers commented on the issue: https://github.com/apache/spark/pull/18229 @srowen, you're right. I changed the strings there as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18229 @mkesselaers looks like there is at least one more test testing the formatted string that will need to be updated --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18229 **[Test build #3783 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3783/testReport)** for PR 18229 at commit [`71c2bbe`](https://github.com/apache/spark/commit/71c2bbe5a42b99e12fce0653a2d23e48eb1358c4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18229 **[Test build #3783 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3783/testReport)** for PR 18229 at commit [`71c2bbe`](https://github.com/apache/spark/commit/71c2bbe5a42b99e12fce0653a2d23e48eb1358c4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user mkesselaers commented on the issue: https://github.com/apache/spark/pull/18229 @srowen , I changed the variable names for `bytesToString`. As far as I see, this should be everything. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user mkesselaers commented on the issue: https://github.com/apache/spark/pull/18229 @srowen , I propose to rewrite the variable names of bytesToString as well, to be consistent. Other occurences of KB can be found in the ivy-report.xsl and in the ByteSuffixes of JavaUtils. Since there seems to be a lot of logic behind the JavaUtils, I would not change this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user mkesselaers commented on the issue: https://github.com/apache/spark/pull/18229 OK, the tests are fixed to take this into account. BytesToString, now correctly output "KiB" instead of "KB" and utils.js now also calculates the correct value. There is still a small difference between BytesToString and FormatBytes, because the first only goes towards a higher size when it is at least 2 MiB, fe. Should we adapt this as well? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18229 Those tests are wrong, in that 200 bytes is not "1953.1 KB". It is "1953.1 KiB". And so on. The numbers should stay the same to make sure behavior doesn't actually change, but the labels should be fixed. The `utils.js` correctly computed MB, etc, but, it should compute MiB and label it as such, for consistency. This was the original issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user mkesselaers commented on the issue:
https://github.com/apache/spark/pull/18229
@srowen , my apologies, I must have misunderstood you.
I changed the calculation to be MebiBytes, but afterwards saw in the tests
that the calculation was already correct
` test("bytesToString") {
assert(Utils.bytesToString(10) === "10.0 B")
assert(Utils.bytesToString(1500) === "1500.0 B")
assert(Utils.bytesToString(200) === "1953.1 KB")
assert(Utils.bytesToString(2097152) === "2.0 MB")
assert(Utils.bytesToString(2306867) === "2.2 MB")
assert(Utils.bytesToString(5368709120L) === "5.0 GB")
assert(Utils.bytesToString(5L * (1L << 40)) === "5.0 TB")
assert(Utils.bytesToString(5L * (1L << 50)) === "5.0 PB")
assert(Utils.bytesToString(5L * (1L << 60)) === "5.0 EB")
assert(Utils.bytesToString(BigInt(1L << 11) * (1L << 60)) === "2.36E+21
B")
}`
Can you please help me with the one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18229 No, this is wrong. The function is still computing megabytes, etc. Did you see my comment in JIRA? there is more to this change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18229: [SPARK-20691][CORE] Difference between Storage Memory as...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18229 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
