[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[dragos]

Github user dragos commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-170863194
  
The discussions started on Nov. 19 2015, "Removing the Mesos fine-grained 
mode". but I only have it in my inbox. I'm sure there's some archive somewhere, 
but I don't have a link at hand.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[dragos]

Github user dragos commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-170512561
  
Discussions on the dev mailing list.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[blbradley]

Github user blbradley commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-170614673
  
@dragos Link?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[blbradley]

Github user blbradley commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-167848940
  
@dragos Where can you see that fine-grained mode is slated for removal? All 
I see is #9795.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[dragos]

Github user dragos closed the pull request at:

https://github.com/apache/spark/pull/9027


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[dragos]

Github user dragos commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-165827968
  
Thanks @tnachen, but since this didn't make it in 1.6, and fine-grained 
mode is slated for removal, I don't think it makes sense to work on it anymore.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[tnachen]

Github user tnachen commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-165336426
  
@dragos just left some comments on this PR, sorry I wasn't able to take a 
deep look earlier. I think there are some open issues that needs to be 
addressed first. Take a look and let me know what you think.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[tnachen]

Github user tnachen commented on a diff in the pull request:

https://github.com/apache/spark/pull/9027#discussion_r47868921
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
 ---
@@ -239,44 +250,69 @@ private[spark] class MesosSchedulerBackend(
   
.setRefuseSeconds(rejectOfferDurationForUnmetConstraints).build())
   }
 
-  // Of the matching constraints, see which ones give us enough memory 
and cores
-  val (usableOffers, unUsableOffers) = 
offersMatchingConstraints.partition { o =>
-val mem = getResource(o.getResourcesList, "mem")
-val cpus = getResource(o.getResourcesList, "cpus")
-val slaveId = o.getSlaveId.getValue
-val offerAttributes = toAttributeMap(o.getAttributesList)
-
-// check offers for
-//  1. Memory requirements
-//  2. CPU requirements - need at least 1 for executor, 1 for task
-val meetsMemoryRequirements = mem >= calculateTotalMemory(sc)
-val meetsCPURequirements = cpus >= (mesosExecutorCores + 
scheduler.CPUS_PER_TASK)
-val meetsRequirements =
-  (meetsMemoryRequirements && meetsCPURequirements) ||
+// Of the matching constraints, see which ones give us enough memory 
and cores
+val (usableOffers, unUsableOffers) = 
offersMatchingConstraints.partition { o =>
+  val mem = getResource(o.getResourcesList, "mem")
+  val cpus = getResource(o.getResourcesList, "cpus")
+  val slaveId = o.getSlaveId.getValue
+  val offerAttributes = toAttributeMap(o.getAttributesList)
+
+  // check offers for
+  //  1. Memory requirements
+  //  2. CPU requirements - need at least 1 for executor, 1 for task
+  val meetsMemoryRequirements = mem >= calculateTotalMemory(sc)
+  val meetsCPURequirements = cpus >= (mesosExecutorCores + 
scheduler.CPUS_PER_TASK)
--- End diff --

I think technically this is not correct (which exists in code before, not 
your change). We can still launch new tasks if there is already an executor 
running without mesosExecutoreCores. Probably fix this in the future.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[tnachen]

Github user tnachen commented on a diff in the pull request:

https://github.com/apache/spark/pull/9027#discussion_r47868971
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
 ---
@@ -239,44 +250,69 @@ private[spark] class MesosSchedulerBackend(
   
.setRefuseSeconds(rejectOfferDurationForUnmetConstraints).build())
   }
 
-  // Of the matching constraints, see which ones give us enough memory 
and cores
-  val (usableOffers, unUsableOffers) = 
offersMatchingConstraints.partition { o =>
-val mem = getResource(o.getResourcesList, "mem")
-val cpus = getResource(o.getResourcesList, "cpus")
-val slaveId = o.getSlaveId.getValue
-val offerAttributes = toAttributeMap(o.getAttributesList)
-
-// check offers for
-//  1. Memory requirements
-//  2. CPU requirements - need at least 1 for executor, 1 for task
-val meetsMemoryRequirements = mem >= calculateTotalMemory(sc)
-val meetsCPURequirements = cpus >= (mesosExecutorCores + 
scheduler.CPUS_PER_TASK)
-val meetsRequirements =
-  (meetsMemoryRequirements && meetsCPURequirements) ||
+// Of the matching constraints, see which ones give us enough memory 
and cores
+val (usableOffers, unUsableOffers) = 
offersMatchingConstraints.partition { o =>
+  val mem = getResource(o.getResourcesList, "mem")
+  val cpus = getResource(o.getResourcesList, "cpus")
+  val slaveId = o.getSlaveId.getValue
+  val offerAttributes = toAttributeMap(o.getAttributesList)
+
+  // check offers for
+  //  1. Memory requirements
+  //  2. CPU requirements - need at least 1 for executor, 1 for task
+  val meetsMemoryRequirements = mem >= calculateTotalMemory(sc)
+  val meetsCPURequirements = cpus >= (mesosExecutorCores + 
scheduler.CPUS_PER_TASK)
+  val meetsRequirements =
+(meetsMemoryRequirements && meetsCPURequirements) ||
   (slaveIdToExecutorInfo.contains(slaveId) && cpus >= 
scheduler.CPUS_PER_TASK)
-val debugstr = if (meetsRequirements) "Accepting" else "Declining"
-logDebug(s"$debugstr offer: ${o.getId.getValue} with attributes: "
-  + s"$offerAttributes mem: $mem cpu: $cpus")
+  val debugstr = if (meetsRequirements) "Accepting" else "Declining"
+  logDebug(s"$debugstr offer: ${o.getId.getValue} with attributes: "
+   + s"$offerAttributes mem: $mem cpu: $cpus")
+
+  meetsRequirements
+}
+
+// Decline offers we ruled out immediately
+unUsableOffers.foreach(o => d.declineOffer(o.getId))
+
+var availableCores = Math.max(0, maxCores - totalCoresAcquired)
 
-meetsRequirements
+val workerOffers = (for (o <- usableOffers) yield {
+  val coresInOffer = getResource(o.getResourcesList, "cpus").toInt
+  val extraCores = if 
(slaveIdToExecutorInfo.contains(o.getSlaveId.getValue)) {
+0D
+  } else {
+// If the Mesos executor has not been started on this slave yet, 
set aside a few
+// cores for the Mesos executor by offering fewer cores to the 
Spark executor
+mesosExecutorCores
   }
 
-  // Decline offers we ruled out immediately
-  unUsableOffers.foreach(o => d.declineOffer(o.getId))
+  // the cores we can offer for tasks on workers should not exceed 
neither availableCores
+  // nor cores in the current offer, after accounting for non-task 
cores
+  val taskCores = Math.min(availableCores - extraCores, coresInOffer - 
extraCores)
 
-  val workerOffers = usableOffers.map { o =>
-val cpus = if 
(slaveIdToExecutorInfo.contains(o.getSlaveId.getValue)) {
-  getResource(o.getResourcesList, "cpus").toInt
-} else {
-  // If the Mesos executor has not been started on this slave yet, 
set aside a few
-  // cores for the Mesos executor by offering fewer cores to the 
Spark executor
-  (getResource(o.getResourcesList, "cpus") - 
mesosExecutorCores).toInt
-}
-new WorkerOffer(
+  if (taskCores > 0) {
+availableCores -= taskCores + extraCores
+Option(new WorkerOffer(
   o.getSlaveId.getValue,
   o.getHostname,
-  cpus)
+  taskCores.toInt))
+  } else {
+None
   }
+}).flatten
+
+(usableOffers, workerOffers)
--- End diff --

What happens if we hit the limit and don't run any more tasks on the 
remaining usableOffers? We need to decline those too right?


---
If your project is set up for it, you can reply to this email and have your
reply 

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[tnachen]

Github user tnachen commented on a diff in the pull request:

https://github.com/apache/spark/pull/9027#discussion_r47869013
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
 ---
@@ -304,20 +340,25 @@ private[spark] class MesosSchedulerBackend(
 mesosTasks.getOrElseUpdate(slaveId, new 
JArrayList[MesosTaskInfo])
   .add(mesosTask)
 slaveIdToResources(slaveId) = remainingResources
+
+totalCoresAcquired += getResource(mesosTask.getResourcesList, 
"cpus")
   }
 }
 
   // Reply to the offers
   val filters = Filters.newBuilder().setRefuseSeconds(1).build() // 
TODO: lower timeout?
 
-  mesosTasks.foreach { case (slaveId, tasks) =>
-slaveIdToWorkerOffer.get(slaveId).foreach(o =>
-  
listenerBus.post(SparkListenerExecutorAdded(System.currentTimeMillis(), slaveId,
-// TODO: Add support for log urls for Mesos
-new ExecutorInfo(o.host, o.cores, Map.empty)))
-)
-logTrace(s"Launching Mesos tasks on slave '$slaveId', 
tasks:\n${getTasksSummary(tasks)}")
-
d.launchTasks(Collections.singleton(slaveIdToOffer(slaveId).getId), tasks, 
filters)
+  mesosTasks.foreach {
+case (slaveId, tasks) =>
+  // add the cores reserved for each Mesos executor (one per slave)
+  totalCoresAcquired += 
getResource(tasks.get(0).getExecutor.getResourcesList, "cpus")
--- End diff --

We should only account for executor resources once per slave, not for every 
task


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[tnachen]

Github user tnachen commented on a diff in the pull request:

https://github.com/apache/spark/pull/9027#discussion_r47865499
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
 ---
@@ -210,12 +216,18 @@ private[spark] class MesosSchedulerBackend(
   }
 
   /**
-   * Method called by Mesos to offer resources on slaves. We respond by 
asking our active task sets
-   * for tasks in order of priority. We fill each node with tasks in a 
round-robin manner so that
-   * tasks are balanced across the cluster.
+   * Return the usable Mesos offers and corresponding WorkerOffers.
+   *
+   * This method declines Mesos offers that don't meet minimum cpu, memory 
or attribute
+   * requirements.
+   *
+   * @param d Mesos SchedulerDriver to decline offers
+   * @param offers Mesos offers to be considered
+   * @return a pair of Mesos offers and corresponding WorkerOffer that can 
be used by the
+   * fine-grained scheduler.
*/
-  override def resourceOffers(d: SchedulerDriver, offers: JList[Offer]) {
-inClassLoader() {
+  private[spark] def usableWorkerOffers(d: SchedulerDriver,
--- End diff --

I think we've consistently put the function parameters that are longer than 
100 in newlines so far, can you move `d: SchedulerDriver` to the next line?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-164607585
  
@dragos can you fix the style here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-156528766
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45878/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-156528763
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[dragos]

Github user dragos commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-156522730
  
@tnachen can you please have a look? I addressed all your points from #8671 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-156527655
  
**[Test build #45878 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45878/consoleFull)**
 for PR 9027 at commit 
[`68cee1c`](https://github.com/apache/spark/commit/68cee1c5c701257a51e7b1f0edff98e56fcfe2ee).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-156528753
  
**[Test build #45878 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45878/consoleFull)**
 for PR 9027 at commit 
[`68cee1c`](https://github.com/apache/spark/commit/68cee1c5c701257a51e7b1f0edff98e56fcfe2ee).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`public class JavaGradientBoostingClassificationExample `\n  * `public class 
JavaGradientBoostingRegressionExample `\n  * `public class 
JavaRandomForestClassificationExample `\n  * `public class 
JavaRandomForestRegressionExample `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[dragos]

Github user dragos commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-153369861
  
Up to date with master, which has a fix in #9282 that was blocking people 
from testing this one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-153371544
  
**[Test build #44926 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44926/consoleFull)**
 for PR 9027 at commit 
[`960755d`](https://github.com/apache/spark/commit/960755d98fad95d77a3a4173c3e21aff3d01a72f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-153406048
  
**[Test build #44926 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44926/consoleFull)**
 for PR 9027 at commit 
[`960755d`](https://github.com/apache/spark/commit/960755d98fad95d77a3a4173c3e21aff3d01a72f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`case class Exchange(`\n  * `class CoalescedPartitioner(val parent: 
Partitioner, val partitionStartIndices: Array[Int])`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-153406569
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44926/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-153406559
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-153370796
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-153370776
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[nraychaudhuri]

Github user nraychaudhuri commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-149896275
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[dragos]

Github user dragos closed the pull request at:

https://github.com/apache/spark/pull/8671


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-146545648
  
  [Test build #43394 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43394/consoleFull)
 for   PR 9027 at commit 
[`5817d68`](https://github.com/apache/spark/commit/5817d68836fed014841afb20676b23e09fcd96dd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-146544834
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[dragos]

GitHub user dragos opened a pull request:

https://github.com/apache/spark/pull/9027

[SPARK-7874][MESOS] Don’t allocate more than spark.cores.max in 
fine-grained mode.

Currently only coarse-grained mode observes `spark.cores.max`. The 
fine-grained mode should follow the same limit, and not go above the defined 
maximum cores.

There's some duplication of logic between the coarse-grained and 
fine-grained mode. I created 
[SPARK-10444](https://issues.apache.org/jira/browse/SPARK-10444) to fix that.

Rework of #8671 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dragos/spark 
issue/mesos/fine-grained-maxCores-wip

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9027.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9027


commit f37d3e202df0de90ae7eed7cba2abb2dfba79f76
Author: Iulian Dragos 
Date:   2015-09-08T11:49:14Z

Don’t allocate more than spark.cores.max in fine-grained mode.

commit 5817d68836fed014841afb20676b23e09fcd96dd
Author: Iulian Dragos 
Date:   2015-09-17T14:30:21Z

Updated tests and reformulated the condition to make it clearer that we 
can’t go negative.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[dragos]

Github user dragos commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-146543767
  
@tnachen @andrewor14 please have a look


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-146544863
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-146578396
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43394/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-146578224
  
  [Test build #43394 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43394/console)
 for   PR 9027 at commit 
[`5817d68`](https://github.com/apache/spark/commit/5817d68836fed014841afb20676b23e09fcd96dd).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9027#issuecomment-146578393
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[dragos]

Github user dragos commented on a diff in the pull request:

https://github.com/apache/spark/pull/8671#discussion_r39605193
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
 ---
@@ -202,55 +207,86 @@ private[spark] class MesosSchedulerBackend(
   }
 
   /**
-   * Method called by Mesos to offer resources on slaves. We respond by 
asking our active task sets
-   * for tasks in order of priority. We fill each node with tasks in a 
round-robin manner so that
-   * tasks are balanced across the cluster.
+   * Return the usable Mesos offers and corresponding WorkerOffers.
+   *
+   * This method declines Mesos offers that don't meet minimum cpu, memory 
or attribute
+   * requirements.
+   *
+   * @param d Mesos SchedulerDriver to decline offers
+   * @param offers Mesos offers to be considered
+   * @return a pair of Mesos offers and corresponding WorkerOffer that can 
be used by the
+   * fine-grained scheduler.
*/
-  override def resourceOffers(d: SchedulerDriver, offers: JList[Offer]) {
-inClassLoader() {
-  // Fail-fast on offers we know will be rejected
-  val (usableOffers, unUsableOffers) = offers.asScala.partition { o =>
-val mem = getResource(o.getResourcesList, "mem")
-val cpus = getResource(o.getResourcesList, "cpus")
-val slaveId = o.getSlaveId.getValue
-val offerAttributes = toAttributeMap(o.getAttributesList)
-
-// check if all constraints are satisfield
-//  1. Attribute constraints
-//  2. Memory requirements
-//  3. CPU requirements - need at least 1 for executor, 1 for task
-val meetsConstraints = 
matchesAttributeRequirements(slaveOfferConstraints, offerAttributes)
-val meetsMemoryRequirements = mem >= calculateTotalMemory(sc)
-val meetsCPURequirements = cpus >= (mesosExecutorCores + 
scheduler.CPUS_PER_TASK)
-
-val meetsRequirements =
-  (meetsConstraints && meetsMemoryRequirements && 
meetsCPURequirements) ||
+  private[spark] def usableWorkerOffers(d: SchedulerDriver,
+offers: JList[Offer]): (Seq[Protos.Offer], Seq[WorkerOffer]) = {
+// Fail-fast on offers we know will be rejected
+val (usableOffers, unUsableOffers) = offers.asScala.partition { o =>
+  val mem = getResource(o.getResourcesList, "mem")
+  val cpus = getResource(o.getResourcesList, "cpus")
+  val slaveId = o.getSlaveId.getValue
+  val offerAttributes = toAttributeMap(o.getAttributesList)
+
+  // check if all constraints are satisfield
+  //  1. Attribute constraints
+  //  2. Memory requirements
+  //  3. CPU requirements - need at least 1 for executor, 1 for task
+  val meetsConstraints = 
matchesAttributeRequirements(slaveOfferConstraints, offerAttributes)
+  val meetsMemoryRequirements = mem >= calculateTotalMemory(sc)
+  val meetsCPURequirements = cpus >= (mesosExecutorCores + 
scheduler.CPUS_PER_TASK)
+
+  val meetsRequirements =
+(meetsConstraints && meetsMemoryRequirements && 
meetsCPURequirements) ||
   (slaveIdToExecutorInfo.contains(slaveId) && cpus >= 
scheduler.CPUS_PER_TASK)
 
-// add some debug messaging
-val debugstr = if (meetsRequirements) "Accepting" else "Declining"
-val id = o.getId.getValue
-logDebug(s"$debugstr offer: $id with attributes: $offerAttributes 
mem: $mem cpu: $cpus")
+  // add some debug messaging
+  val debugstr = if (meetsRequirements) "Accepting" else "Declining"
+  val id = o.getId.getValue
+  logDebug(s"$debugstr offer: $id with attributes: $offerAttributes 
mem: $mem cpu: $cpus")
+
+  meetsRequirements
+}
+
+// Decline offers we ruled out immediately
+unUsableOffers.foreach(o => d.declineOffer(o.getId))
+
+var availableCores = Math.max(0, maxCores - totalCoresAcquired)
 
-meetsRequirements
+val workerOffers = (for (o <- usableOffers) yield {
+  val coresInOffer = getResource(o.getResourcesList, "cpus")
+  val cores = if 
(slaveIdToExecutorInfo.contains(o.getSlaveId.getValue)) {
+coresInOffer.toInt
+  } else {
+// If the Mesos executor has not been started on this slave yet, 
set aside a few
+// cores for the Mesos executor by offering fewer cores to the 
Spark executor
+availableCores -= mesosExecutorCores
+(coresInOffer - mesosExecutorCores).toInt
   }
 
-  // Decline offers we ruled out immediately
-  unUsableOffers.foreach(o => d.declineOffer(o.getId))
+  // check that we can still acquire cpus
+  val actualCores = 

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[tnachen]

Github user tnachen commented on a diff in the pull request:

https://github.com/apache/spark/pull/8671#discussion_r39575837
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
 ---
@@ -202,55 +207,86 @@ private[spark] class MesosSchedulerBackend(
   }
 
   /**
-   * Method called by Mesos to offer resources on slaves. We respond by 
asking our active task sets
-   * for tasks in order of priority. We fill each node with tasks in a 
round-robin manner so that
-   * tasks are balanced across the cluster.
+   * Return the usable Mesos offers and corresponding WorkerOffers.
+   *
+   * This method declines Mesos offers that don't meet minimum cpu, memory 
or attribute
+   * requirements.
+   *
+   * @param d Mesos SchedulerDriver to decline offers
+   * @param offers Mesos offers to be considered
+   * @return a pair of Mesos offers and corresponding WorkerOffer that can 
be used by the
+   * fine-grained scheduler.
*/
-  override def resourceOffers(d: SchedulerDriver, offers: JList[Offer]) {
-inClassLoader() {
-  // Fail-fast on offers we know will be rejected
-  val (usableOffers, unUsableOffers) = offers.asScala.partition { o =>
-val mem = getResource(o.getResourcesList, "mem")
-val cpus = getResource(o.getResourcesList, "cpus")
-val slaveId = o.getSlaveId.getValue
-val offerAttributes = toAttributeMap(o.getAttributesList)
-
-// check if all constraints are satisfield
-//  1. Attribute constraints
-//  2. Memory requirements
-//  3. CPU requirements - need at least 1 for executor, 1 for task
-val meetsConstraints = 
matchesAttributeRequirements(slaveOfferConstraints, offerAttributes)
-val meetsMemoryRequirements = mem >= calculateTotalMemory(sc)
-val meetsCPURequirements = cpus >= (mesosExecutorCores + 
scheduler.CPUS_PER_TASK)
-
-val meetsRequirements =
-  (meetsConstraints && meetsMemoryRequirements && 
meetsCPURequirements) ||
+  private[spark] def usableWorkerOffers(d: SchedulerDriver,
+offers: JList[Offer]): (Seq[Protos.Offer], Seq[WorkerOffer]) = {
+// Fail-fast on offers we know will be rejected
+val (usableOffers, unUsableOffers) = offers.asScala.partition { o =>
+  val mem = getResource(o.getResourcesList, "mem")
+  val cpus = getResource(o.getResourcesList, "cpus")
+  val slaveId = o.getSlaveId.getValue
+  val offerAttributes = toAttributeMap(o.getAttributesList)
+
+  // check if all constraints are satisfield
+  //  1. Attribute constraints
+  //  2. Memory requirements
+  //  3. CPU requirements - need at least 1 for executor, 1 for task
+  val meetsConstraints = 
matchesAttributeRequirements(slaveOfferConstraints, offerAttributes)
+  val meetsMemoryRequirements = mem >= calculateTotalMemory(sc)
+  val meetsCPURequirements = cpus >= (mesosExecutorCores + 
scheduler.CPUS_PER_TASK)
+
+  val meetsRequirements =
+(meetsConstraints && meetsMemoryRequirements && 
meetsCPURequirements) ||
   (slaveIdToExecutorInfo.contains(slaveId) && cpus >= 
scheduler.CPUS_PER_TASK)
 
-// add some debug messaging
-val debugstr = if (meetsRequirements) "Accepting" else "Declining"
-val id = o.getId.getValue
-logDebug(s"$debugstr offer: $id with attributes: $offerAttributes 
mem: $mem cpu: $cpus")
+  // add some debug messaging
+  val debugstr = if (meetsRequirements) "Accepting" else "Declining"
+  val id = o.getId.getValue
+  logDebug(s"$debugstr offer: $id with attributes: $offerAttributes 
mem: $mem cpu: $cpus")
+
+  meetsRequirements
+}
+
+// Decline offers we ruled out immediately
+unUsableOffers.foreach(o => d.declineOffer(o.getId))
+
+var availableCores = Math.max(0, maxCores - totalCoresAcquired)
 
-meetsRequirements
+val workerOffers = (for (o <- usableOffers) yield {
+  val coresInOffer = getResource(o.getResourcesList, "cpus")
+  val cores = if 
(slaveIdToExecutorInfo.contains(o.getSlaveId.getValue)) {
+coresInOffer.toInt
+  } else {
+// If the Mesos executor has not been started on this slave yet, 
set aside a few
+// cores for the Mesos executor by offering fewer cores to the 
Spark executor
+availableCores -= mesosExecutorCores
+(coresInOffer - mesosExecutorCores).toInt
   }
 
-  // Decline offers we ruled out immediately
-  unUsableOffers.foreach(o => d.declineOffer(o.getId))
+  // check that we can still acquire cpus
+  val actualCores = 

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[dragos]

Github user dragos commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-140327944
  
@tnachen @andrewor14 would you have some time to have a look at this one?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[dragos]

Github user dragos commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-139135979
  
/cc @tnachen @andrewor14 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[dragos]

GitHub user dragos opened a pull request:

https://github.com/apache/spark/pull/8671

[SPARK-7874][MESOS] Don’t allocate more than spark.cores.max in 
fine-grained mode.

Currently only coarse-grained mode observes `spark.cores.max`. The 
fine-grained mode should follow the same limit, and not go above the defined 
maximum cores.

There's some duplication of logic between the coarse-grained and 
fine-grained mode. I created 
[SPARK-10444](https://issues.apache.org/jira/browse/SPARK-10444) to fix that.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dragos/spark issue/mesos/fine-grained-maxCores

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8671.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8671


commit deca1d316bc7e65ce3a8221f7926f7b597e5
Author: Iulian Dragos 
Date:   2015-09-08T11:49:14Z

Don’t allocate more than spark.cores.max in fine-grained mode.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-138940045
  
  [Test build #42206 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42206/console)
 for   PR 8671 at commit 
[`deca1d3`](https://github.com/apache/spark/commit/deca1d316bc7e65ce3a8221f7926f7b597e5).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-138943753
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-138940049
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-138944431
  
  [Test build #42207 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42207/consoleFull)
 for   PR 8671 at commit 
[`6f748c5`](https://github.com/apache/spark/commit/6f748c59bd8043de136e4d52f5f377efb5ca6994).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-138937812
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-138937837
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-138940051
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42206/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-138939525
  
  [Test build #42206 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42206/consoleFull)
 for   PR 8671 at commit 
[`deca1d3`](https://github.com/apache/spark/commit/deca1d316bc7e65ce3a8221f7926f7b597e5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-138943717
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-138988135
  
  [Test build #42207 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42207/console)
 for   PR 8671 at commit 
[`6f748c5`](https://github.com/apache/spark/commit/6f748c59bd8043de136e4d52f5f377efb5ca6994).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-138988273
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7874][MESOS] Don’t allocate more than...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8671#issuecomment-138988278
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42207/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org