[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/5063 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-94112146 LGTM I'm merging this finally thanks everyone. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93898567 @andrewor14 I've fixed what you issue. Please review and merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93882452 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30448/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93872085 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93882446 [Test build #30448 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30448/consoleFull) for PR 5063 at commit [`9238d6e`](https://github.com/apache/spark/commit/9238d6e581e676e92e6cad95bc1e6fda1c4c031c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93872612 [Test build #30448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30448/consoleFull) for PR 5063 at commit [`9238d6e`](https://github.com/apache/spark/commit/9238d6e581e676e92e6cad95bc1e6fda1c4c031c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93751671 [Test build #30418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30418/consoleFull) for PR 5063 at commit [`2d41241`](https://github.com/apache/spark/commit/2d4124110d36e04cb004688f1a7832ee0be5481b). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93751686 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30418/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93748213 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30419/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93748187 [Test build #30419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30419/consoleFull) for PR 5063 at commit [`9238d6e`](https://github.com/apache/spark/commit/9238d6e581e676e92e6cad95bc1e6fda1c4c031c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93732843 [Test build #30419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30419/consoleFull) for PR 5063 at commit [`9238d6e`](https://github.com/apache/spark/commit/9238d6e581e676e92e6cad95bc1e6fda1c4c031c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93731580 I've rebased it from current master at first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93730968 [Test build #30418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30418/consoleFull) for PR 5063 at commit [`2d41241`](https://github.com/apache/spark/commit/2d4124110d36e04cb004688f1a7832ee0be5481b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93206397 @andrewor14 Thanks for overall reviewing. I'll handle what you issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93132743 I talked to @sryza and @tnachen offline about the potential sources of confusion here. It seems that this code used to mistakenly use `spark.task.cpus` as the number of cores to give Mesos executors, which is incorrect but happens to be fine because `spark.task.cpus` defaults to 1. I left a more detailed comment about why the existing name of the config introduces another potential source of confusion. In general, when making changes in the Spark on Mesos part of the code base, we should be explicit about which kind of task and executor we are referring to, since these terms unfortunately have overloaded meanings at the intersection of these two projects. @jongyoul The intended change in behavior here LGTM. Once you address the wording / naming comments I left I will go ahead and merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r28386662 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,16 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.executor.cores + 1.0 + +Set the amount of cores, which can be a floating point number, to request for running a Mesos executor. +The setting controls the number of cores allocated to the executor not for use by tasks. By default, +executor will use the amount of cores even though no task is running on an executor. This setting is +only used for Mesos fine-grained mode. --- End diff -- This is a little verbose and ambiguous. Can we rephrase this as: ``` (Fine-grained mode only) Number of cores to give each Mesos executor. This does not include the cores used to run the Spark tasks themselves. In other words, even if no Spark task is being run, the Mesos executor will occupy the number of cores configured here. ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5063#discussion_r28386441
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
---
@@ -233,9 +234,7 @@ private[spark] class MesosSchedulerBackend(
getResource(o.getResourcesList, "cpus").toInt
} else {
// If the executor doesn't exist yet, subtract CPU for executor
--- End diff --
Perhaps we can elaborate on this comment a little:
```
// If the Mesos executor has not been started on this slave yet, set aside
a few
// cores for the Mesos executor by subtracting this from the number that
would
// have been granted to the Spark executor
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r28386348 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,16 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.executor.cores --- End diff -- Similarly, the naming of this config is misleading. In the rest of Spark we use the dot notation to denote hierarchy. For instance, we have `spark.yarn.am.waitTime`, which falls under the `spark.yarn.*` namespace. In this case, people might interpret this as the number of cores to give the Spark executor, since this config falls under the `spark.mesos.*` namespace and elsewhere "executor" means Spark executors. So we should probably rename this to `spark.mesos.mesosExecutor.cores`. Unfortunately, this needs to start with `spark.mesos.*` to keep it consistent with other Mesos-specific configs, and it needs to be `mesosExecutor` because otherwise people may interpret it as Spark executor cores instead. It's a little more verbose but much less ambiguous. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5063#discussion_r28385856
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
---
@@ -67,6 +67,8 @@ private[spark] class MesosSchedulerBackend(
// The listener bus to publish executor added/removed events.
val listenerBus = sc.listenerBus
+
+ val executorCores = sc.conf.getDouble("spark.mesos.executor.cores", 1)
--- End diff --
This variable should be renamed `mesosExecutorCores`. In Spark, whenever we
see the word "executor" we automatically think of Spark executors. Just having
`executorCores` here is somewhat misleading.
Also, I know the rest of this file doesn't do this, but can you make this
`private[mesos]` so it's clear that it's only read within this package?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r28381413 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -220,10 +222,9 @@ private[spark] class MesosSchedulerBackend( val mem = getResource(o.getResourcesList, "mem") val cpus = getResource(o.getResourcesList, "cpus") val slaveId = o.getSlaveId.getValue -// TODO(pwendell): Should below be 1 + scheduler.CPUS_PER_TASK? (mem >= MemoryUtils.calculateTotalMemory(sc) && // need at least 1 for executor, 1 for task --- End diff -- @andrewor14 I've missed it. It doesn't need --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93105439 What @sryza is correct. spark.mesos.executor.cores is different than spark.executor.cores, as it's only the cpu cores used for the mesos executor, which is used to launch Spark executors and those executors are tied to tasks and have its own cpu allocation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93058391 My understanding based on the discussion here is that `spark.mesos.executor.cores` is the number of cores reserved by an executor *not* for use in running tasks. So if `spark.mesos.executor.cores` is 1, `spark.task.cpus` is 2, and 3 tasks are running, then a total of 7 cores are being occupied. The primary use case for setting it to a number different than 1 is that Mesos allows values that are smaller than 1. So, when running multiple executors per node, someone might set it to .1 in order to avoid sitting on a bunch of the node's cores. Did you look at the documentation for the new property? If this wasn't clear, then we should probably update the doc with a better explanation or link to relevant Mesos doc. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93056104 @sryza @tnachen just so I understand, doesn't `spark.mesos.executor.cores` refer to the number of cores used by the Spark executor launched to run a given task? Why is it necessary to differentiate between the number of cores used by the executor for running the task, and the number of cores used by it to do other things? What are the differences in semantics between this and `spark.executor.cores` exactly? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r28365258 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -220,10 +222,9 @@ private[spark] class MesosSchedulerBackend( val mem = getResource(o.getResourcesList, "mem") val cpus = getResource(o.getResourcesList, "cpus") val slaveId = o.getSlaveId.getValue -// TODO(pwendell): Should below be 1 + scheduler.CPUS_PER_TASK? (mem >= MemoryUtils.calculateTotalMemory(sc) && // need at least 1 for executor, 1 for task --- End diff -- is this comment still valid? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93040635 @andrewor14 this was a concern I had as well, and there's a long discussion above. The reason for not using `spark.executor.cores` is that the property introduced here actually refers to something different. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93011001 Should we reuse the existing `spark.executor.cores` property? It would be good to make all the cluster managers use a consistent configuration than to introduce a separate `spark.mesos.executor.cores`. This is currently under discussion in #4027 (for Mesos) and #731 (for standalone mode). cc'ing @pwendell --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-90266435 @andrewor14 do you think you can merge this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-88293005 @sryza Could you please merge this PR for spark 1.3.1? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-87791403 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-87544727 @sryza I hope this will be a final review :-) I'm sorry for making a typo again and again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-87532194 [Test build #29377 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29377/consoleFull) for PR 5063 at commit [`afcdda5`](https://github.com/apache/spark/commit/afcdda56966c21d35bbacdf7318e225d818e3e98). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-87532197 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29377/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-87513975 [Test build #29377 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29377/consoleFull) for PR 5063 at commit [`afcdda5`](https://github.com/apache/spark/commit/afcdda56966c21d35bbacdf7318e225d818e3e98). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86973622 One minor nit and otherwise LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r27302728 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,16 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.executor.cores + 1.0 + +Set the amount of cores, which can be a floating point number, to request for running a mesos executor. --- End diff -- Mesos should be capitalized. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86841550 @sryza Review it again, please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86841449 [Test build #29285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29285/consoleFull) for PR 5063 at commit [`8c3c065`](https://github.com/apache/spark/commit/8c3c0652a5cbb78921324bb3bd0a4ea4216c167f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86841450 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29285/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86824172 [Test build #29285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29285/consoleFull) for PR 5063 at commit [`8c3c065`](https://github.com/apache/spark/commit/8c3c0652a5cbb78921324bb3bd0a4ea4216c167f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r27275980 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,14 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.executor.cores + 1.0 + +The setting, which can be a floating point number, controls the number of cores allocated --- End diff -- Ok, I see. It's not a big problem to edit docs. :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86812055 [Test build #29272 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29272/consoleFull) for PR 5063 at commit [`223aed9`](https://github.com/apache/spark/commit/223aed924c19c3862e2f1b29d9c8c98d3fc86e54). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86812076 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29272/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r27274938 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,14 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.executor.cores + 1.0 + +The setting, which can be a floating point number, controls the number of cores allocated --- End diff -- Sorry, maybe I should have been more clear, but I meant for this text to go in *in addition* to the paragraph you originally had above it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86796016 [Test build #29272 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29272/consoleFull) for PR 5063 at commit [`223aed9`](https://github.com/apache/spark/commit/223aed924c19c3862e2f1b29d9c8c98d3fc86e54). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86795209 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86365655 [Test build #29215 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29215/consoleFull) for PR 5063 at commit [`223aed9`](https://github.com/apache/spark/commit/223aed924c19c3862e2f1b29d9c8c98d3fc86e54). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86365666 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29215/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86357920 [Test build #29215 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29215/consoleFull) for PR 5063 at commit [`223aed9`](https://github.com/apache/spark/commit/223aed924c19c3862e2f1b29d9c8c98d3fc86e54). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86357586 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86357482 @sryza Who Do you know help this build issue? I just edited docs only. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86356704 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29214/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86356693 [Test build #29214 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29214/consoleFull) for PR 5063 at commit [`223aed9`](https://github.com/apache/spark/commit/223aed924c19c3862e2f1b29d9c8c98d3fc86e54). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class UnregisterApplication(appId: String)` * `class RegexTokenizer extends UnaryTransformer[String, Seq[String], RegexTokenizer] ` * `case class Data(weight: Double, mu: Vector, sigma: Matrix)` * `implicit class DslSymbol(sym: Symbol) extends ImplicitAttribute ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86354447 [Test build #29212 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29212/consoleFull) for PR 5063 at commit [`5c82f96`](https://github.com/apache/spark/commit/5c82f960427a1f29de59a8feed7dee3070048a3c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86354455 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29212/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86345756 [Test build #29214 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29214/consoleFull) for PR 5063 at commit [`223aed9`](https://github.com/apache/spark/commit/223aed924c19c3862e2f1b29d9c8c98d3fc86e54). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86345441 I've rebased from master. Don't mind that I have a mistake to write a commit log which I've worked another issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86344972 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86343836 [Test build #29212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29212/consoleFull) for PR 5063 at commit [`5c82f96`](https://github.com/apache/spark/commit/5c82f960427a1f29de59a8feed7dee3070048a3c). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86341349 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29204/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86341311 [Test build #29204 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29204/consoleFull) for PR 5063 at commit [`49811dd`](https://github.com/apache/spark/commit/49811dde03dd1d1315249f4cc0d3aa5a331f7b39). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86327593 [Test build #29204 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29204/consoleFull) for PR 5063 at commit [`49811dd`](https://github.com/apache/spark/commit/49811dde03dd1d1315249f4cc0d3aa5a331f7b39). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86327232 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86326206 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29202/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86326197 [Test build #29202 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29202/consoleFull) for PR 5063 at commit [`49811dd`](https://github.com/apache/spark/commit/49811dde03dd1d1315249f4cc0d3aa5a331f7b39). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86309543 [Test build #29202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29202/consoleFull) for PR 5063 at commit [`49811dd`](https://github.com/apache/spark/commit/49811dde03dd1d1315249f4cc0d3aa5a331f7b39). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86155273 Had some doc comments, but after they're fixed this LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r27149333 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,14 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.executor.cores + 1.0 + +Set the amount of cores to request for running a mesos executor. This setting is only used for Mesos fine-grained mode. +By default, executor will use the amount of cores even though no task is running on an executor. --- End diff -- I would change this to "The setting, which can be a floating point number, controls the number of cores allocated to the executor not for use by tasks." --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r27149062 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,14 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.executor.cores + 1.0 + +Set the amount of cores to request for running a mesos executor. This setting is only used for Mesos fine-grained mode. --- End diff -- Lines should be wrapped liked the other properties. Also, Mesos should be capitalized in both places. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86113945 @jongyoul looks good to me thanks :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85965536 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29159/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85965524 [Test build #29159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29159/consoleFull) for PR 5063 at commit [`a8bc7ee`](https://github.com/apache/spark/commit/a8bc7ee6cd822573f57af1100cc01aa4e63423a0). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85921782 [Test build #29159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29159/consoleFull) for PR 5063 at commit [`a8bc7ee`](https://github.com/apache/spark/commit/a8bc7ee6cd822573f57af1100cc01aa4e63423a0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85920174 @tnachen @elyast I've updated it to support fractional number of cores for executors. Please review this PR. I've tested my cluster. My setting is ``` AVA_HOME=/app/jdk HADOOP_HOME=/app/hdfs HADOOP_CONF_DIR=/app/hdfs/conf MESOS_NATIVE_JAVA_LIBRARY=/app/mesos-0.21.0/lib/libmesos.so SPARK_EXECUTOR_MEMORY=5G spark-1.4.0-SNAPSHOT-bin-2.5.0-cdh5.3.0/bin/spark-shell --master mesos://zk://dicc-m001:2181,dicc-m002:2181,dicc-m003:2181/mesos-0.21.0 --conf spark.executor.uri=hdfs:///app/spark/spark-1.4.0-SNAPSHOT-bin-2.5.0-cdh5.3.0.tgz --conf spark.io.compression.codec=lzf --conf spark.task.cpus=5 --conf spark.mesos.executor.cores=0.1 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85596738 Yes let's keep CPUS_PER_TASK as is, since it's shared among different things. Executor cores I think can still be a double that is only used in MesosSchedulerBackend. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85594000 @jongyoul @tnachen imho its fine if CPU_PER_TASK will remain Integers, however if your job is less CPU intensive, that might be beneficial to optimize it, although I don't have strong opinion on that. Maybe we just proceed with this one and create new jira ticket on CPU_PER_TASK? if there would be demand for that then we could proceed with that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85500449 @elyast @tnachen Do you think `CPUS_PER_TASK` also support fractional value? If it's not, I may be support executorCores as fractional value without huge changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85336463 @jongyoul I can try to test it out with allocating CPU to 0, but cannot promise it will work, otherwise it's just doesn't make much sense imho, I'd discuss with project committers if they think changing from int to double is such a big deal. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85317877 @tnachen I see. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85096806 Fractional is definitely supported, since it's just cpu shares in the end. We should make it a double --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85096699 @sryza When creating a Mesos Task, one usually define the resources required for the execution of the task and the resources required to run the Mesos executor. Again the executor role is initiate executing the task and report task statuses, but can do anything else if it's a custom executor provided by the user. (You can skip defining executor where Mesos provides a default one and also add a default resource padding for the default one). In Spark fine-grain mode we do have a custom executor in org.apache.spark.executor.MesosExecutorBackend, and cores assigned is just for running this executor alone which is running one per slave per app (it can run mulitple Spark tasks). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85002672 @sryza you can request from Mesos fraction of CPU, however I haven't realized that we have wrong type in this patch, we should change it to Double instead of Int. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84814176 @elyast can you explain what you mean in a little more detail? Are you saying that the patch somehow enables users to request fractional cores? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84801569 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28971/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84801541 [Test build #28971 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28971/consoleFull) for PR 5063 at commit [`f6e78db`](https://github.com/apache/spark/commit/f6e78dbce628e2cb2183e4780a40f8de72c5ddf0). * This patch **passes all tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84801446 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28974/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84801414 [Test build #28974 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28974/consoleFull) for PR 5063 at commit [`9f160f5`](https://github.com/apache/spark/commit/9f160f5798d757841f0534f3d6c4795b591c422b). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84770282 [Test build #28974 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28974/consoleFull) for PR 5063 at commit [`9f160f5`](https://github.com/apache/spark/commit/9f160f5798d757841f0534f3d6c4795b591c422b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84769305 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84769168 I rebased this from master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84762169 [Test build #28971 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28971/consoleFull) for PR 5063 at commit [`f6e78db`](https://github.com/apache/spark/commit/f6e78dbce628e2cb2183e4780a40f8de72c5ddf0). * This patch **does not merge cleanly**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84761674 Jekins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84733162 @sryza I don't think it actually need more than a single core, the issue is you cannot give less than 1 CPU. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84712739 I see. So frameworkCores isn't ideal because "framework" already has a meaning in the Mesos world. Also, "executor" already has some inherent ambiguity because Mesos uses it differently than Spark. So `spark.mesos.executor.cores` sounds reasonable to me. I'm still somewhat confused by the need for such a property (and this might still be stemming from me not understanding about how Spark Mesos integration works). If my understanding is correct, the cores included in `spark.mesos.executor.cores` are never made available to tasks. If that's the case, when would the Mesos executor ever need more than a single core? Does anything the executor is doing require much CPU? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84587905 **[Test build #28959 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28959/consoleFull)** for PR 5063 at commit [`f6e78db`](https://github.com/apache/spark/commit/f6e78dbce628e2cb2183e4780a40f8de72c5ddf0) after a configured wait of `120m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84587909 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28959/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84564916 [Test build #28959 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28959/consoleFull) for PR 5063 at commit [`f6e78db`](https://github.com/apache/spark/commit/f6e78dbce628e2cb2183e4780a40f8de72c5ddf0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84272183 @sryza I believe this setting is a bit unique for Mesos. I think it's better to clarify the terminology here, of what does framework and executor means for Mesos. In mesos a scheduler is the process which registers with Master to receives offers, and use these offers to launch tasks through the master and eventually the slave. An executor is responsible for actually launching a task and reporting task statuses that is launched on the slaves, which either is a custom executor that fine-grain mode uses or the default mesos executor that coarse-grain mode uses. And a framework is referring to the overall scheduler + executor. So in the Mesos fine-grain case, we have an custom executor that is only launched once per slave, and that running executor takes up some cpu/mem itself. IMO I think calling it spark.mesos.executor.cores is fine enough, I wouldn't put framework in there as framework is not the right terminology. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84258060 @elyast It's enough to explain why we can set this property. @sryza This feature exists only in Mesos mode now. How about setting this property specified in Mesos now, and changing it later if any of cluster will use this property? I don't thinks It's not generic now, but It is needed in Mesos absolutely. I recommend the property name of `spark.mesos.executor.freameworkCores`, or `spark.mesos.executor.cores` looks fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user elyast commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84059146 Hi @sryza @jongyoul, To give an illustration of this, let's say I have 10 nodes, 64 cores each, lets say 10 streaming jobs are running with 1 minute window (so every minute they will launch tasks) and lets say those 10 streaming jobs apparently run 10 executors each. Lets say processing each minute takes 10 seconds. At the end of the day you will end up with 100 cores reservation on Mesos that will be there as long as streaming applications are running. (and the executor most probably are not doing any useful work most of the time). Currently there is no way to tweak how many cores it is given to executor, u may even think u can just assign 0.1 CPU for better utilization of resources. Let me know what do u think --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
