[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-82031855 @tnachen @elyast I made a new issue about configuring mesos executor cores. https://issues.apache.org/jira/browse/SPARK-6350 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-82104142 cool thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-80798498 @elyast Thanks for interesting this PR, which was about resources of cores and memory. I misunderstood how mesos works specially in memory side, so I closed this PR. However, I agree with you that executor cores are sometime too much. @tnachen we cannot adjust memory issue, but, changing executor cores is meaningful and should fix that executor has same cores with CPUS_PER_TASK initially. See TODO(pwendell) in MesosSchedulerBackend.scala. I want to fix this TODO to extract a configuration parameter. What do you think of it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-80862357 I think making it a configurable parameter sounda reasonable to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-78868580 @elyast yes you are correct it is only applicable per Spark app. It is entirely possible to make executor cpu less than 1 (as it's based on shares), but it's not possible for now to share mesos executor across apps. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-79046678 Sure its totally fine not to share, but at least it should be possible to configure allocation. Allocating 1 CPU per executor may just too much, obviously it depends how cpu intensive is his work, but I guess @mateiz know that much better than me --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-78799120 One comment, however if you run multiple Spark applications even tough executor-id == slave-id, multiple executors can be started on the same host. (And every one of them will consume 1 CPU without scheduling any tasks). This can be painful when you want to run multiple streaming applications on Mesos in fine grained mode, because each streaming driver's executors will consume 1 CPU...  Screen shots illustrate situation on single slave, when there are two executors running for 2 different Spark applications (one is streaming app, second one is Zeppelin), and as u can see there 0 active tasks the consumption of CPU is 2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71767996 @mateiz That's a sample screenshot.  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71762746 Sorry, what was the problem here? Executors *do* use memory, in fact they use all of the memory of that JVM (the only reason we assign memory to tasks is that Mesos didn't support tasks with 0 memory). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71768903 Basically I don't understand what your patch is fixing. What's an example of config settings that gave the wrong allocation before, and what will they give now? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71765405 @mateiz We agree with one executor and multi task is intended behaviour. In this situation, MesosScheduler offers CPUS_PER_TASK resources to executor when we launch separate task. If we launch two tasks, we offers 3 * CPUS_PER_TASK(= 1 for executor and 2 for tasks) for running only two tasks. @pwendell thinks that's too much resources and It's enough for executor to have one cores. In my PR, I enable to set executor's cores. In memory, we just offered memory to executor only. If we launch two tasks again, we offers 1 * calculateTotalMemory(sc) to all tasks. I think we offer one executor memory and two task memories. I agree that executor uses memory by itself, but we fix amount of those value --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71765614 Thus, I believe that executor has executor cores and executor memory setting on ExecutorInfo, and task has its own cores and memories setting on TaskInfo while launching per task. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71768825 Sorry, I'm not sure I understand about the memory. There should be the same amount of memory for each executor. More executors means more total memory. But the way that was being calculated before is fine, what's wrong with it? For CPUs, I understand wanting to start executors off at 0 CPUs and just having 1 per task. I actually thought we did that initially, not sure why it's 1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71770002 Sorry, I've shaw you my configuration. my configuraion is 5G for SPARK_EXECUTOR_MEMORY and 5 for spark.task.cpus. In my screenshot, we launch two tasks on the same machine. Don't you think It's good to offer task memory twice? My PR gives correct resource management information to mesos' master. For CPUs, I don't know proper value of executor cpus, but not CPUS_TASK_CPUS. Recommend this value, please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71771833 @jongyul sorry didn't get to finish reviewing the PR, and I agree with matei that in spark usage of mesos it doesn't make sense to give tasks memory, as we share the same executor that is kept running. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71772544 I don't know the behaviour in coarse-grained mode, but in fine-grained mode, we use multiple JVM for running tasks. we run spark-class by launcher. This means we launch JVM by running per task. Am I wrong? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71773477 I believed that when we launch mesos driver launchTasks, container run the command `bin/spark-class` everytime running task. And in my qna email for mesos, @tnachen answers that one container run multiple command simultaneously. And my some tests show two tasks runs simutaneously because they write a same log file at the same time. And my digging codes results no limit to launch task on a mesos container. However, @mateiz told me that one executor only runs a single JVM and launch a single task at any time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71774383 @jongyoul So an executor can only launch one task at a time, but can have multiple tasks running simultaneously as you mentioned. It doesn't matter if they're all part of the same launchTasks message or seperate, as long as the framework and executor id are the same it will be launched in the same executor. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71775404 @tnachen @mateiz So sorry for taking up a lot of time. I've found that only one executor as a process runs at any time, and I understand executor can have multiple tasks at the same time. I've believed each executor is launched separately when driver launchTasks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71773647 If you read the fine-grained mode source code, you'll notice that Spark is using the slave id as the executor id, which is what we discussed on the mesos mailing list, that the executor will be re-used if all tasks reuse the same executor id. Therefore, it's only launching one executor per slave, and if the executor dies Mesos will relaunch it when the task asks for it again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71773877 @tnachen Yes, I fully understand reusing executor while a framework is alive. However, we launch two task on a same executor? What you've answered is they are launched at the same time, isn't it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71770662 Right, as I said, it doesn't make sense to offer task memory twice. Each executor is a *single* JVM, and JVMs cannot scale their memory up and down. The executor's memory is set to the same value that we configure that JVM with, with `-Xmx`. There's no way to make tasks use more memory than that, no matter how many tasks are running on there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71776099 I'll close this PR. It's wrong approach. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul closed the pull request at: https://github.com/apache/spark/pull/4170 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71411958 /cc @mateiz Could you please review this PR which is about offering resources to executor and task. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71166652 [Test build #26015 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26015/consoleFull) for PR 4170 at commit [`d714e8b`](https://github.com/apache/spark/commit/d714e8bb6b699e5ec2a315df65cee0f4cf7765e5). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71174326 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26015/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71174318 [Test build #26015 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26015/consoleFull) for PR 4170 at commit [`d714e8b`](https://github.com/apache/spark/commit/d714e8bb6b699e5ec2a315df65cee0f4cf7765e5). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71159960 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26003/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71159955 [Test build #26003 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26003/consoleFull) for PR 4170 at commit [`a28b666`](https://github.com/apache/spark/commit/a28b66684236bfb31fce3b9fcaaf8e33e166b5b5). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
GitHub user jongyoul opened a pull request: https://github.com/apache/spark/pull/4170 [SPARK-5376][Mesos] MesosExecutor should have correct resources - Divided task and executor resources - Added `spark.mesos.executor.cpus` and fixed docs You can merge this pull request into a Git repository by running: $ git pull https://github.com/jongyoul/spark SPARK-5376 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4170.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4170 commit 71703c808027b54970367af7d26f0d151b6a8003 Author: Jongyoul Lee jongy...@gmail.com Date: 2015-01-23T05:46:25Z [SPARK-5376][Mesos] MesosExecutor should have correct resources - Divided task and executor resources - Added `spark.mesos.executor.cpus` and fixed docs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71150639 /cc @tnachen @pwendell This PR is about @pwendell 's todo. Review this, please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71150832 [Test build #25997 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25997/consoleFull) for PR 4170 at commit [`71703c8`](https://github.com/apache/spark/commit/71703c808027b54970367af7d26f0d151b6a8003). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71151118 [Test build #25998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25998/consoleFull) for PR 4170 at commit [`f655eee`](https://github.com/apache/spark/commit/f655eee2bd3d871e6b50fbd9dd3a826662abca30). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/4170#discussion_r23433902 --- Diff: docs/running-on-mesos.md --- @@ -226,6 +226,13 @@ See the [configuration page](configuration.html) for information on Spark config The final total amount of memory allocated is the maximum value between executor memory plus memoryOverhead, and overhead fraction (1.07) plus the executor memory. /td /tr +tr + tdcodespark.mesos.executor.cpus/code/td + td1/td + td +The amount of cores that Mesos executor will request additionally for running executor itself. --- End diff -- Please use the same description in the other doc, and the change the value too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/4170#discussion_r23433897 --- Diff: docs/configuration.md --- @@ -341,6 +341,13 @@ Apart from these, the following properties are also available, and may be useful `spark.mesos.executor.memoryOverhead` or 7% of `spark.executor.memory`. /td /tr +tr + tdcodespark.mesos.executor.cpus/code/td + td1/td + td +This value is the amount of cores so that executor is running itself. --- End diff -- Spark code has been referring cpus to cores, so I assume you want to name this spark.mesos.executor.cores I think we should re-word this, to something like: The amount of cores to request for running the mesos executor. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on a diff in the pull request: https://github.com/apache/spark/pull/4170#discussion_r23433986 --- Diff: docs/configuration.md --- @@ -341,6 +341,13 @@ Apart from these, the following properties are also available, and may be useful `spark.mesos.executor.memoryOverhead` or 7% of `spark.executor.memory`. /td /tr +tr + tdcodespark.mesos.executor.cpus/code/td + td1/td + td +This value is the amount of cores so that executor is running itself. --- End diff -- I agree. I'm also confused between cpus and cores. I think `cores` is more proper term. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71157359 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25999/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71157354 [Test build #25999 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25999/consoleFull) for PR 4170 at commit [`9054535`](https://github.com/apache/spark/commit/90545351ac53e054fc64c8229f97c19d82cb60aa). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71155131 [Test build #26003 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26003/consoleFull) for PR 4170 at commit [`a28b666`](https://github.com/apache/spark/commit/a28b66684236bfb31fce3b9fcaaf8e33e166b5b5). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71153592 [Test build #25999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25999/consoleFull) for PR 4170 at commit [`9054535`](https://github.com/apache/spark/commit/90545351ac53e054fc64c8229f97c19d82cb60aa). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71153844 [Test build #25997 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25997/consoleFull) for PR 4170 at commit [`71703c8`](https://github.com/apache/spark/commit/71703c808027b54970367af7d26f0d151b6a8003). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71153847 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25997/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71153845 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25998/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71153842 [Test build #25998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25998/consoleFull) for PR 4170 at commit [`f655eee`](https://github.com/apache/spark/commit/f655eee2bd3d871e6b50fbd9dd3a826662abca30). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
