[jira] [Commented] (AMATERASU-24) Refactor Spark out of Amaterasu executor to it's own project
[ https://issues.apache.org/jira/browse/AMATERASU-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535683#comment-16535683 ] ASF GitHub Bot commented on AMATERASU-24: - roadan commented on issue #30: AMATERASU-24 Refactor Spark out of Amaterasu executor to it's own pro… URL: https://github.com/apache/incubator-amaterasu/pull/30#issuecomment-403198711 Tested on mesos and yarn This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor Spark out of Amaterasu executor to it's own project > > > Key: AMATERASU-24 > URL: https://issues.apache.org/jira/browse/AMATERASU-24 > Project: AMATERASU > Issue Type: Improvement >Reporter: Yaniv Rodenski >Assignee: Arun Manivannan >Priority: Major > Fix For: 0.2.1-incubating > > > The Spark framework is a part of the Amaterasu executor and leader, it needs > to be under it own project under a new frameworks folder -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMATERASU-24) Refactor Spark out of Amaterasu executor to it's own project
[
https://issues.apache.org/jira/browse/AMATERASU-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535684#comment-16535684
]
ASF GitHub Bot commented on AMATERASU-24:
-
roadan closed pull request #30: AMATERASU-24 Refactor Spark out of Amaterasu
executor to it's own pro…
URL: https://github.com/apache/incubator-amaterasu/pull/30
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.gitignore b/.gitignore
index fabb847..dbb51cb 100755
--- a/.gitignore
+++ b/.gitignore
@@ -48,3 +48,5 @@ bin/
amaterasu-executor/
project/project/
executor/spark-warehouse/
+repo
+repo/**
diff --git a/build.gradle b/build.gradle
index 0f11347..00e44ea 100644
--- a/build.gradle
+++ b/build.gradle
@@ -25,10 +25,6 @@ allprojects {
version '0.2.0-incubating-rc4'
}
-project(':leader')
-project(':common')
-project(':executor')
-
task copyLeagalFiles(type: Copy) {
from "./DISCLAIMER", "./LICENSE", "./NOTICE"
into "${buildDir}/amaterasu"
diff --git
a/common/src/main/scala/org/apache/amaterasu/common/execution/actions/Notifier.scala
b/common/src/main/scala/org/apache/amaterasu/common/execution/actions/Notifier.scala
index 8a44019..fe69260 100755
---
a/common/src/main/scala/org/apache/amaterasu/common/execution/actions/Notifier.scala
+++
b/common/src/main/scala/org/apache/amaterasu/common/execution/actions/Notifier.scala
@@ -16,9 +16,9 @@
*/
package org.apache.amaterasu.common.execution.actions
-import NotificationLevel.NotificationLevel
-import NotificationType.NotificationType
import com.fasterxml.jackson.annotation.JsonProperty
+import
org.apache.amaterasu.common.execution.actions.NotificationLevel.NotificationLevel
+import
org.apache.amaterasu.common.execution.actions.NotificationType.NotificationType
abstract class Notifier {
diff --git a/executor/build.gradle b/executor/build.gradle
index 21bc2b0..09e269c 100644
--- a/executor/build.gradle
+++ b/executor/build.gradle
@@ -54,7 +54,6 @@ dependencies {
compile group: 'org.scala-lang', name: 'scala-library', version: '2.11.8'
compile group: 'org.scala-lang', name: 'scala-reflect', version: '2.11.8'
-compile group: 'org.scala-lang', name: 'scala-compiler', version: '2.11.8'
compile group: 'io.netty', name: 'netty-all', version: '4.0.42.Final'
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.5'
compile group: 'org.apache.maven', name: 'maven-core', version: '3.0.5'
@@ -75,18 +74,7 @@ dependencies {
compile project(':common')
compile project(':amaterasu-sdk')
-//runtime dependency for spark
-provided('org.apache.spark:spark-repl_2.11:2.2.1')
-provided('org.apache.spark:spark-core_2.11:2.2.1')
-
-testCompile project(':common')
-testCompile "gradle.plugin.com.github.maiflai:gradle-scalatest:0.14"
-testRuntime 'org.pegdown:pegdown:1.1.0'
-testCompile 'junit:junit:4.11'
-testCompile 'org.scalatest:scalatest_2.11:3.0.2'
-testCompile 'org.scala-lang:scala-library:2.11.8'
-testCompile('org.apache.spark:spark-repl_2.11:2.2.1')
-testCompile group: 'org.slf4j', name: 'slf4j-api', version: '1.7.9'
+
}
diff --git
a/executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/MesosActionsExecutor.scala
b/executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/MesosActionsExecutor.scala
index 9ab75be..90c2001 100755
---
a/executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/MesosActionsExecutor.scala
+++
b/executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/MesosActionsExecutor.scala
@@ -26,7 +26,6 @@ import
org.apache.amaterasu.executor.common.executors.ProvidersFactory
import org.apache.mesos.Protos._
import org.apache.mesos.protobuf.ByteString
import org.apache.mesos.{Executor, ExecutorDriver, MesosExecutorDriver}
-import org.apache.spark.SparkContext
import scala.collection.JavaConverters._
import scala.concurrent.ExecutionContext.Implicits.global
@@ -37,7 +36,6 @@ class MesosActionsExecutor extends Executor with Logging {
var master: String = _
var executorDriver: ExecutorDriver = _
- var sc: SparkContext = _
var jobId: String = _
var actionName: String = _
// var sparkScalaRunner: SparkScalaRunner = _
@@ -83,7 +81,7 @@ class MesosActionsExecutor extends Executor with Logging {
notifier = new MesosNotifier(driver)
notifier.info(s"Executor ${executorInfo.getExecutorId.getValue}
registered")
val outStream = new ByteArrayOutputStream()
-providersFactory = ProvidersFactory(data, jobId, outStream, notifier,
executorInfo.getExecutorId.getValue, hostName, propFile =
"./amaterasu.properties")
+providersFactory =
[jira] [Commented] (AMATERASU-24) Refactor Spark out of Amaterasu executor to it's own project
[ https://issues.apache.org/jira/browse/AMATERASU-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529002#comment-16529002 ] ASF GitHub Bot commented on AMATERASU-24: - arunma opened a new pull request #30: AMATERASU-24 Refactor Spark out of Amaterasu executor to it's own pro… URL: https://github.com/apache/incubator-amaterasu/pull/30 …ject (Master) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor Spark out of Amaterasu executor to it's own project > > > Key: AMATERASU-24 > URL: https://issues.apache.org/jira/browse/AMATERASU-24 > Project: AMATERASU > Issue Type: Improvement >Reporter: Yaniv Rodenski >Assignee: Arun Manivannan >Priority: Major > Fix For: 0.2.1-incubating > > > The Spark framework is a part of the Amaterasu executor and leader, it needs > to be under it own project under a new frameworks folder -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMATERASU-24) Refactor Spark out of Amaterasu executor to it's own project
[
https://issues.apache.org/jira/browse/AMATERASU-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528932#comment-16528932
]
ASF GitHub Bot commented on AMATERASU-24:
-
roadan closed pull request #27: AMATERASU-24 Refactor Spark out of Amaterasu
executor to it's own project
URL: https://github.com/apache/incubator-amaterasu/pull/27
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/build.gradle b/build.gradle
index 0f11347..00e44ea 100644
--- a/build.gradle
+++ b/build.gradle
@@ -25,10 +25,6 @@ allprojects {
version '0.2.0-incubating-rc4'
}
-project(':leader')
-project(':common')
-project(':executor')
-
task copyLeagalFiles(type: Copy) {
from "./DISCLAIMER", "./LICENSE", "./NOTICE"
into "${buildDir}/amaterasu"
diff --git
a/common/src/main/scala/org/apache/amaterasu/common/execution/actions/Notifier.scala
b/common/src/main/scala/org/apache/amaterasu/common/execution/actions/Notifier.scala
index 8a44019..fe69260 100755
---
a/common/src/main/scala/org/apache/amaterasu/common/execution/actions/Notifier.scala
+++
b/common/src/main/scala/org/apache/amaterasu/common/execution/actions/Notifier.scala
@@ -16,9 +16,9 @@
*/
package org.apache.amaterasu.common.execution.actions
-import NotificationLevel.NotificationLevel
-import NotificationType.NotificationType
import com.fasterxml.jackson.annotation.JsonProperty
+import
org.apache.amaterasu.common.execution.actions.NotificationLevel.NotificationLevel
+import
org.apache.amaterasu.common.execution.actions.NotificationType.NotificationType
abstract class Notifier {
diff --git a/executor/build.gradle b/executor/build.gradle
index 21bc2b0..09e269c 100644
--- a/executor/build.gradle
+++ b/executor/build.gradle
@@ -54,7 +54,6 @@ dependencies {
compile group: 'org.scala-lang', name: 'scala-library', version: '2.11.8'
compile group: 'org.scala-lang', name: 'scala-reflect', version: '2.11.8'
-compile group: 'org.scala-lang', name: 'scala-compiler', version: '2.11.8'
compile group: 'io.netty', name: 'netty-all', version: '4.0.42.Final'
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.5'
compile group: 'org.apache.maven', name: 'maven-core', version: '3.0.5'
@@ -75,18 +74,7 @@ dependencies {
compile project(':common')
compile project(':amaterasu-sdk')
-//runtime dependency for spark
-provided('org.apache.spark:spark-repl_2.11:2.2.1')
-provided('org.apache.spark:spark-core_2.11:2.2.1')
-
-testCompile project(':common')
-testCompile "gradle.plugin.com.github.maiflai:gradle-scalatest:0.14"
-testRuntime 'org.pegdown:pegdown:1.1.0'
-testCompile 'junit:junit:4.11'
-testCompile 'org.scalatest:scalatest_2.11:3.0.2'
-testCompile 'org.scala-lang:scala-library:2.11.8'
-testCompile('org.apache.spark:spark-repl_2.11:2.2.1')
-testCompile group: 'org.slf4j', name: 'slf4j-api', version: '1.7.9'
+
}
diff --git
a/executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/MesosActionsExecutor.scala
b/executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/MesosActionsExecutor.scala
index 9ab75be..90c2001 100755
---
a/executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/MesosActionsExecutor.scala
+++
b/executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/MesosActionsExecutor.scala
@@ -26,7 +26,6 @@ import
org.apache.amaterasu.executor.common.executors.ProvidersFactory
import org.apache.mesos.Protos._
import org.apache.mesos.protobuf.ByteString
import org.apache.mesos.{Executor, ExecutorDriver, MesosExecutorDriver}
-import org.apache.spark.SparkContext
import scala.collection.JavaConverters._
import scala.concurrent.ExecutionContext.Implicits.global
@@ -37,7 +36,6 @@ class MesosActionsExecutor extends Executor with Logging {
var master: String = _
var executorDriver: ExecutorDriver = _
- var sc: SparkContext = _
var jobId: String = _
var actionName: String = _
// var sparkScalaRunner: SparkScalaRunner = _
@@ -83,7 +81,7 @@ class MesosActionsExecutor extends Executor with Logging {
notifier = new MesosNotifier(driver)
notifier.info(s"Executor ${executorInfo.getExecutorId.getValue}
registered")
val outStream = new ByteArrayOutputStream()
-providersFactory = ProvidersFactory(data, jobId, outStream, notifier,
executorInfo.getExecutorId.getValue, hostName, propFile =
"./amaterasu.properties")
+providersFactory = ProvidersFactory(data, jobId, outStream, notifier,
executorInfo.getExecutorId.getValue, hostName, "./amaterasu.properties")
}
diff --git
[jira] [Commented] (AMATERASU-24) Refactor Spark out of Amaterasu executor to it's own project
[ https://issues.apache.org/jira/browse/AMATERASU-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528931#comment-16528931 ] ASF GitHub Bot commented on AMATERASU-24: - roadan commented on issue #27: AMATERASU-24 Refactor Spark out of Amaterasu executor to it's own project URL: https://github.com/apache/incubator-amaterasu/pull/27#issuecomment-401573891 Tested on YARN and Mesos This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor Spark out of Amaterasu executor to it's own project > > > Key: AMATERASU-24 > URL: https://issues.apache.org/jira/browse/AMATERASU-24 > Project: AMATERASU > Issue Type: Improvement >Reporter: Yaniv Rodenski >Assignee: Arun Manivannan >Priority: Major > Fix For: 0.2.1-incubating > > > The Spark framework is a part of the Amaterasu executor and leader, it needs > to be under it own project under a new frameworks folder -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMATERASU-24) Refactor Spark out of Amaterasu executor to it's own project
[ https://issues.apache.org/jira/browse/AMATERASU-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506816#comment-16506816 ] ASF GitHub Bot commented on AMATERASU-24: - arunma commented on issue #24: AMATERASU-24 Refactor Spark out of Amaterasu executor to it's own pro… URL: https://github.com/apache/incubator-amaterasu/pull/24#issuecomment-395939568 @roadan @eyalbenivri @nadav-har-tzvi The kicking off the executor needs some work but appreciate if you could have a quick look and let me know if the changes are on the right track. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor Spark out of Amaterasu executor to it's own project > > > Key: AMATERASU-24 > URL: https://issues.apache.org/jira/browse/AMATERASU-24 > Project: AMATERASU > Issue Type: Improvement >Reporter: Yaniv Rodenski >Assignee: Arun Manivannan >Priority: Major > Fix For: 0.2.1-incubating > > > The Spark framework is a part of the Amaterasu executor and leader, it needs > to be under it own project under a new frameworks folder -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMATERASU-24) Refactor Spark out of Amaterasu executor to it's own project
[ https://issues.apache.org/jira/browse/AMATERASU-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506815#comment-16506815 ] ASF GitHub Bot commented on AMATERASU-24: - arunma opened a new pull request #24: AMATERASU-24 Refactor Spark out of Amaterasu executor to it's own pro… URL: https://github.com/apache/incubator-amaterasu/pull/24 …ject This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor Spark out of Amaterasu executor to it's own project > > > Key: AMATERASU-24 > URL: https://issues.apache.org/jira/browse/AMATERASU-24 > Project: AMATERASU > Issue Type: Improvement >Reporter: Yaniv Rodenski >Assignee: Arun Manivannan >Priority: Major > Fix For: 0.2.1-incubating > > > The Spark framework is a part of the Amaterasu executor and leader, it needs > to be under it own project under a new frameworks folder -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMATERASU-24) Refactor Spark out of Amaterasu executor to it's own project
[ https://issues.apache.org/jira/browse/AMATERASU-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491973#comment-16491973 ] ASF GitHub Bot commented on AMATERASU-24: - arunma opened a new pull request #22: AMATERASU-24 Refactor Spark out of Amaterasu executor to it's own pro… URL: https://github.com/apache/incubator-amaterasu/pull/22 …ject This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor Spark out of Amaterasu executor to it's own project > > > Key: AMATERASU-24 > URL: https://issues.apache.org/jira/browse/AMATERASU-24 > Project: AMATERASU > Issue Type: Improvement >Reporter: Yaniv Rodenski >Priority: Major > Fix For: 0.2.1-incubating > > > The Spark framework is a part of the Amaterasu executor and leader, it needs > to be under it own project under a new frameworks folder -- This message was sent by Atlassian JIRA (v7.6.3#76005)
