[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

2016-08-20 Thread srowen 
Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/14637#discussion_r75576449
  
--- Diff: mesos/pom.xml ---
@@ -0,0 +1,109 @@
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+  4.0.0
+  
+org.apache.spark
+spark-parent_2.11
+2.1.0-SNAPSHOT
+../pom.xml
+  
+
+  spark-mesos_2.11
+  jar
+  Spark Project Mesos
+  
+mesos
+1.0.0
+shaded-protobuf
+  
+
+  
+
+  org.apache.spark
+  spark-core_${scala.binary.version}
+  ${project.version}
+
+
+
+  org.apache.spark
+  spark-core_${scala.binary.version}
+  ${project.version}
+  test-jar
+  test
+
+
+
+  org.apache.mesos
+  mesos
+  ${mesos.version}
+  ${mesos.classifier}
+  
+
+  com.google.protobuf
+  protobuf-java
+
+  
+
+
+
--- End diff --

I don't know that you need these dependencies -- literally don't know 
either way by looking. These are definitely needed? I was wondering whether 
jetty was relevant.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

2016-08-20 Thread jaceklaskowski 
Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14637#discussion_r75574292
  
--- Diff: project/SparkBuild.scala ---
@@ -56,9 +56,9 @@ object BuildCommons {
 "tags", "sketch"
   ).map(ProjectRef(buildLocation, _)) ++ sqlProjects ++ streamingProjects
 
-  val optionallyEnabledProjects@Seq(yarn, java8Tests, sparkGangliaLgpl,
+  val optionallyEnabledProjects@Seq(mesos, yarn, java8Tests, 
sparkGangliaLgpl,
--- End diff --

This is a pattern matching on assignment feature in Scala and the line 
creates one `optionallyEnabledProjects` value for the entire `Seq` of projects 
with respective projects being their own values `mesos` etc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

2016-08-19 Thread mgummelt 
Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14637#discussion_r75563699
  
--- Diff: project/SparkBuild.scala ---
@@ -56,9 +56,9 @@ object BuildCommons {
 "tags", "sketch"
   ).map(ProjectRef(buildLocation, _)) ++ sqlProjects ++ streamingProjects
 
-  val optionallyEnabledProjects@Seq(yarn, java8Tests, sparkGangliaLgpl,
+  val optionallyEnabledProjects@Seq(mesos, yarn, java8Tests, 
sparkGangliaLgpl,
--- End diff --

FYI I'm just pattern matching here to get `-Pmesos` to work with `sbt`.  I 
have no idea what this does.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

2016-08-19 Thread mgummelt 
Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14637#discussion_r75563505
  
--- Diff: mesos/pom.xml ---
@@ -0,0 +1,167 @@
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
--- End diff --

I don't think line length rules apply to XML.  See the other poms for 
examples.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

2016-08-19 Thread mgummelt 
Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14637#discussion_r75547904
  
--- Diff: 
mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterManager.scala
 ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler.cluster
+
+import org.apache.spark.{SparkContext, SparkException}
+import org.apache.spark.scheduler.{ExternalClusterManager, 
SchedulerBackend, TaskScheduler, TaskSchedulerImpl}
+import 
org.apache.spark.scheduler.cluster.mesos.{MesosCoarseGrainedSchedulerBackend, 
MesosFineGrainedSchedulerBackend}
+
+/**
+ * Cluster Manager for creation of Yarn scheduler and backend
+ */
+private[spark] class MesosClusterManager extends ExternalClusterManager {
+  private val MESOS_REGEX = """mesos://(.*)""".r
+
+  override def canCreate(masterURL: String): Boolean = {
+masterURL.startsWith("mesos")
+  }
+
+  override def createTaskScheduler(sc: SparkContext, masterURL: String): 
TaskScheduler = {
+new TaskSchedulerImpl(sc)
+  }
+
+  override def createSchedulerBackend(sc: SparkContext,
+  masterURL: String,
+  scheduler: TaskScheduler): 
SchedulerBackend = {
+val mesosUrl = MESOS_REGEX.findFirstMatchIn(masterURL).get.group(1)
+val coarse = sc.conf.getBoolean("spark.mesos.coarse", defaultValue = 
true)
--- End diff --

Mesos supports coarse-grained and fine-grained, but fine-grained is 
deprecated 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

2016-08-14 Thread mgummelt 
Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14637#discussion_r74706594
  
--- Diff: mesos/pom.xml ---
@@ -0,0 +1,167 @@
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+  4.0.0
+  
+org.apache.spark
+spark-parent_2.11
+2.1.0-SNAPSHOT
+../pom.xml
+  
+
+  spark-mesos_2.11
+  jar
+  Spark Project Mesos
+  
+mesos
+1.0.0
+shaded-protobuf
+  
+
+  
+
+  org.apache.spark
+  spark-core_${scala.binary.version}
+  ${project.version}
+
+
+  org.apache.spark
+  spark-core_${scala.binary.version}
+  ${project.version}
+  test-jar
+  test
+
+
+  org.apache.mesos
+  mesos
+  ${mesos.version}
+  ${mesos.classifier}
+  
+
+  com.google.protobuf
+  protobuf-java
+
+  
+
+
+
+

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

2016-08-14 Thread jaceklaskowski 
Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14637#discussion_r74706568
  
--- Diff: 
mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterManager.scala
 ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler.cluster
+
+import org.apache.spark.{SparkContext, SparkException}
+import org.apache.spark.scheduler.{ExternalClusterManager, 
SchedulerBackend, TaskScheduler, TaskSchedulerImpl}
+import 
org.apache.spark.scheduler.cluster.mesos.{MesosCoarseGrainedSchedulerBackend, 
MesosFineGrainedSchedulerBackend}
+
+/**
+ * Cluster Manager for creation of Yarn scheduler and backend
+ */
+private[spark] class MesosClusterManager extends ExternalClusterManager {
+  private val MESOS_REGEX = """mesos://(.*)""".r
+
+  override def canCreate(masterURL: String): Boolean = {
+masterURL.startsWith("mesos")
+  }
+
+  override def createTaskScheduler(sc: SparkContext, masterURL: String): 
TaskScheduler = {
+new TaskSchedulerImpl(sc)
+  }
+
+  override def createSchedulerBackend(sc: SparkContext,
+  masterURL: String,
+  scheduler: TaskScheduler): 
SchedulerBackend = {
+val mesosUrl = MESOS_REGEX.findFirstMatchIn(masterURL).get.group(1)
+val coarse = sc.conf.getBoolean("spark.mesos.coarse", defaultValue = 
true)
--- End diff --

Oh, I thought Spark supports coarse-grain scheduling only (even for Mesos 
that had fine-grained one too). Will it stay or is this a temporary thing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

2016-08-14 Thread jaceklaskowski 
Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14637#discussion_r74706552
  
--- Diff: 
mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterManager.scala
 ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler.cluster
+
+import org.apache.spark.{SparkContext, SparkException}
+import org.apache.spark.scheduler.{ExternalClusterManager, 
SchedulerBackend, TaskScheduler, TaskSchedulerImpl}
+import 
org.apache.spark.scheduler.cluster.mesos.{MesosCoarseGrainedSchedulerBackend, 
MesosFineGrainedSchedulerBackend}
+
+/**
+ * Cluster Manager for creation of Yarn scheduler and backend
+ */
+private[spark] class MesosClusterManager extends ExternalClusterManager {
+  private val MESOS_REGEX = """mesos://(.*)""".r
+
+  override def canCreate(masterURL: String): Boolean = {
+masterURL.startsWith("mesos")
+  }
+
+  override def createTaskScheduler(sc: SparkContext, masterURL: String): 
TaskScheduler = {
+new TaskSchedulerImpl(sc)
+  }
+
+  override def createSchedulerBackend(sc: SparkContext,
+  masterURL: String,
--- End diff --

I think the convention is to inline 4 spaces only not to align to the first 
parameter.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

2016-08-14 Thread jaceklaskowski 
Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14637#discussion_r74706517
  
--- Diff: mesos/pom.xml ---
@@ -0,0 +1,167 @@
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+  4.0.0
+  
+org.apache.spark
+spark-parent_2.11
+2.1.0-SNAPSHOT
+../pom.xml
+  
+
+  spark-mesos_2.11
+  jar
+  Spark Project Mesos
+  
+mesos
+1.0.0
+shaded-protobuf
+  
+
+  
+
+  org.apache.spark
+  spark-core_${scala.binary.version}
+  ${project.version}
+
+
+  org.apache.spark
+  spark-core_${scala.binary.version}
+  ${project.version}
+  test-jar
+  test
+
+
+  org.apache.mesos
+  mesos
+  ${mesos.version}
+  ${mesos.classifier}
+  
+
+  com.google.protobuf
+  protobuf-java
+
+  
+
+
+
+

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

2016-08-14 Thread jaceklaskowski 
Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14637#discussion_r74706522
  
--- Diff: mesos/pom.xml ---
@@ -0,0 +1,167 @@
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+  4.0.0
+  
+org.apache.spark
+spark-parent_2.11
+2.1.0-SNAPSHOT
+../pom.xml
+  
+
+  spark-mesos_2.11
+  jar
+  Spark Project Mesos
+  
+mesos
+1.0.0
+shaded-protobuf
+  
+
+  
+
+  org.apache.spark
+  spark-core_${scala.binary.version}
+  ${project.version}
+
+
+  org.apache.spark
+  spark-core_${scala.binary.version}
+  ${project.version}
+  test-jar
+  test
+
+
+  org.apache.mesos
+  mesos
+  ${mesos.version}
+  ${mesos.classifier}
+  
+
+  com.google.protobuf
+  protobuf-java
+
+  
+
+
+
+
+
+
+  org.mockito
+  mockito-core
+  test
+
+

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

2016-08-14 Thread jaceklaskowski 
Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14637#discussion_r74706502
  
--- Diff: mesos/pom.xml ---
@@ -0,0 +1,167 @@
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
--- End diff --

Isn't the line too long? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

2016-08-14 Thread jaceklaskowski 
Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14637#discussion_r74706494
  
--- Diff: mesos/pom.xml ---
@@ -0,0 +1,167 @@
+
+

[GitHub] spark pull request #14637: [WIP] [SPARK-16967] move mesos to module

2016-08-14 Thread mgummelt 
GitHub user mgummelt opened a pull request:

https://github.com/apache/spark/pull/14637

[WIP] [SPARK-16967] move mesos to module

## What changes were proposed in this pull request?

Move Mesos code into a mvn module

## How was this patch tested?

unit tests
manually submitting a client mode and cluster mode job
spark/mesos integration test suite



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mesosphere/spark mesos-module

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14637.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14637


commit ef9c4815a98777d5172c491a4243cbd5839bfd2e
Author: Michael Gummelt 
Date:   2016-08-14T17:42:06Z

move mesos to module




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org