[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/5065 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-83238890 OK, I think this should be merged as-is now since it correctly describes the current behavior. You are welcome to open an issue that then changes that behavior to be consistent with YARN, which I imagine is what was intended. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-83030138 @srowen No, Mesos doc is not wrong, and I mean that Mesos also uses MB as same as MiB. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-82895967 Yes MB and MiB are different things. It looks like Mesos says it uses "MB" so I'd match that, I suppose. Are you saying that Mesos doc is wrong? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-82856317 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28786/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-82856285 [Test build #28786 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28786/consoleFull) for PR 5065 at commit [`c5af84c`](https://github.com/apache/spark/commit/c5af84cf29cb221378e96cbdea8c521a9cbcd094). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-82816654 [Test build #28786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28786/consoleFull) for PR 5065 at commit [`c5af84c`](https://github.com/apache/spark/commit/c5af84cf29cb221378e96cbdea8c521a9cbcd094). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-82701925 @srowen I'm sorry not for understanding your opinion fully - I'm not a native english -, do you mean that it's proper to use "MB" instead of "MiB"? I've found Mesos project used that "MB" has meant "MiB". And I'll follow up an unintentional difference in a separate issue, too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-82244546 Yes this needs to be update. There are some small pre-existing problems with this text though, which should be fixed here too. MiB is a mebibyte, but the value is actually in MB, or megabytes. I imagine that a lot of code/text elsewhere maybe talks about MB when it means MiB, but here this should match Mesos docs which indicate that passing 384 will mean 384MB. More importantly, I realized that the logic is slightly different here between Mesos and YARN. In YARN, the overhead is always completely overrideable by setting `spark.yarn.executor.memoryOverhead`. It merely defaults to the greater of 384, or 10% of the executor memory. In the Mesos `MemoryUtils`, yes, it's the max of 10% of executor memory, or `spark.mesos.memoryOverhead` which defaults to 384. I think this is an unintentional difference that should be rectified? right now the caller has no way of overriding the 10% minimum, which doesn't seem right. I'm OK to commit this (I can change "MiB") since it would correctly reflect the logic now, but I'd welcome it if you follow up by reviewing the argument above and look at changing `MemoryUtils` in a separate issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-82228524 [Test build #28713 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28713/consoleFull) for PR 5065 at commit [`dbac1c0`](https://github.com/apache/spark/commit/dbac1c05168bf9c3a0e0e7d10767a5600a6848d1). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-82228598 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28713/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-82193193 @srowen Review this PR, please. This is a second part of SPARK-6085 which you committed. I just fixed a small documentation, so I don't make another issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-82192968 [Test build #28713 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28713/consoleFull) for PR 5065 at commit [`dbac1c0`](https://github.com/apache/spark/commit/dbac1c05168bf9c3a0e0e7d10767a5600a6848d1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
GitHub user jongyoul opened a pull request: https://github.com/apache/spark/pull/5065 SPARK-6085 Part. 2 Increase default value for memory overhead - fixed a description of spark.mesos.executor.memoryOverhead from 7% to 10% - This is a second part of SPARK-6085 You can merge this pull request into a Git repository by running: $ git pull https://github.com/jongyoul/spark SPARK-6085-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/5065.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5065 commit dbac1c05168bf9c3a0e0e7d10767a5600a6848d1 Author: Jongyoul Lee Date: 2015-03-17T07:57:17Z SPARK-6085 Part. 2 Increase default value for memory overhead - fixed a description of spark.mesos.executor.memoryOverhead from 7% to 10% --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
