Podling report this month
Hi guys, we have to report this month: https://wiki.apache.org/incubator/July2018 Can someone start a report template or I do ? Thanks, Regards JB -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com
[jira] [Created] (AMATERASU-43) Make the notifier available for job code
Nadav Har Tzvi created AMATERASU-43: --- Summary: Make the notifier available for job code Key: AMATERASU-43 URL: https://issues.apache.org/jira/browse/AMATERASU-43 Project: AMATERASU Issue Type: Improvement Reporter: Nadav Har Tzvi -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AMATERASU-42) Amaterasu needs to respect Maven classifiers
Nadav Har Tzvi created AMATERASU-42: --- Summary: Amaterasu needs to respect Maven classifiers Key: AMATERASU-42 URL: https://issues.apache.org/jira/browse/AMATERASU-42 Project: AMATERASU Issue Type: Improvement Reporter: Nadav Har Tzvi -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMATERASU-41) Support nested custom configurations in jobs environments in PySpark
[ https://issues.apache.org/jira/browse/AMATERASU-41?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nadav Har Tzvi updated AMATERASU-41: Summary: Support nested custom configurations in jobs environments in PySpark (was: Support nested custom configurations in jobs environments) > Support nested custom configurations in jobs environments in PySpark > > > Key: AMATERASU-41 > URL: https://issues.apache.org/jira/browse/AMATERASU-41 > Project: AMATERASU > Issue Type: Improvement >Reporter: Nadav Har Tzvi >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AMATERASU-41) Support nested custom configurations in jobs environments
Nadav Har Tzvi created AMATERASU-41: --- Summary: Support nested custom configurations in jobs environments Key: AMATERASU-41 URL: https://issues.apache.org/jira/browse/AMATERASU-41 Project: AMATERASU Issue Type: Improvement Reporter: Nadav Har Tzvi -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AMATERASU-40) Change Spark-Scala runner to use spark-submit
Nadav Har Tzvi created AMATERASU-40: --- Summary: Change Spark-Scala runner to use spark-submit Key: AMATERASU-40 URL: https://issues.apache.org/jira/browse/AMATERASU-40 Project: AMATERASU Issue Type: Improvement Reporter: Nadav Har Tzvi -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AMATERASU-39) Support SSH for Git
Nadav Har Tzvi created AMATERASU-39: --- Summary: Support SSH for Git Key: AMATERASU-39 URL: https://issues.apache.org/jira/browse/AMATERASU-39 Project: AMATERASU Issue Type: Task Reporter: Nadav Har Tzvi -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMATERASU-38) Failure to interpret multiline Scala code blocks
[ https://issues.apache.org/jira/browse/AMATERASU-38?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nadav Har Tzvi updated AMATERASU-38: Priority: Blocker (was: Major) > Failure to interpret multiline Scala code blocks > > > Key: AMATERASU-38 > URL: https://issues.apache.org/jira/browse/AMATERASU-38 > Project: AMATERASU > Issue Type: Bug >Reporter: Nadav Har Tzvi >Priority: Blocker > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMATERASU-24) Refactor Spark out of Amaterasu executor to it's own project
[ https://issues.apache.org/jira/browse/AMATERASU-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535683#comment-16535683 ] ASF GitHub Bot commented on AMATERASU-24: - roadan commented on issue #30: AMATERASU-24 Refactor Spark out of Amaterasu executor to it's own pro… URL: https://github.com/apache/incubator-amaterasu/pull/30#issuecomment-403198711 Tested on mesos and yarn This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor Spark out of Amaterasu executor to it's own project > > > Key: AMATERASU-24 > URL: https://issues.apache.org/jira/browse/AMATERASU-24 > Project: AMATERASU > Issue Type: Improvement >Reporter: Yaniv Rodenski >Assignee: Arun Manivannan >Priority: Major > Fix For: 0.2.1-incubating > > > The Spark framework is a part of the Amaterasu executor and leader, it needs > to be under it own project under a new frameworks folder -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMATERASU-24) Refactor Spark out of Amaterasu executor to it's own project
[
https://issues.apache.org/jira/browse/AMATERASU-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535684#comment-16535684
]
ASF GitHub Bot commented on AMATERASU-24:
-
roadan closed pull request #30: AMATERASU-24 Refactor Spark out of Amaterasu
executor to it's own pro…
URL: https://github.com/apache/incubator-amaterasu/pull/30
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.gitignore b/.gitignore
index fabb847..dbb51cb 100755
--- a/.gitignore
+++ b/.gitignore
@@ -48,3 +48,5 @@ bin/
amaterasu-executor/
project/project/
executor/spark-warehouse/
+repo
+repo/**
diff --git a/build.gradle b/build.gradle
index 0f11347..00e44ea 100644
--- a/build.gradle
+++ b/build.gradle
@@ -25,10 +25,6 @@ allprojects {
version '0.2.0-incubating-rc4'
}
-project(':leader')
-project(':common')
-project(':executor')
-
task copyLeagalFiles(type: Copy) {
from "./DISCLAIMER", "./LICENSE", "./NOTICE"
into "${buildDir}/amaterasu"
diff --git
a/common/src/main/scala/org/apache/amaterasu/common/execution/actions/Notifier.scala
b/common/src/main/scala/org/apache/amaterasu/common/execution/actions/Notifier.scala
index 8a44019..fe69260 100755
---
a/common/src/main/scala/org/apache/amaterasu/common/execution/actions/Notifier.scala
+++
b/common/src/main/scala/org/apache/amaterasu/common/execution/actions/Notifier.scala
@@ -16,9 +16,9 @@
*/
package org.apache.amaterasu.common.execution.actions
-import NotificationLevel.NotificationLevel
-import NotificationType.NotificationType
import com.fasterxml.jackson.annotation.JsonProperty
+import
org.apache.amaterasu.common.execution.actions.NotificationLevel.NotificationLevel
+import
org.apache.amaterasu.common.execution.actions.NotificationType.NotificationType
abstract class Notifier {
diff --git a/executor/build.gradle b/executor/build.gradle
index 21bc2b0..09e269c 100644
--- a/executor/build.gradle
+++ b/executor/build.gradle
@@ -54,7 +54,6 @@ dependencies {
compile group: 'org.scala-lang', name: 'scala-library', version: '2.11.8'
compile group: 'org.scala-lang', name: 'scala-reflect', version: '2.11.8'
-compile group: 'org.scala-lang', name: 'scala-compiler', version: '2.11.8'
compile group: 'io.netty', name: 'netty-all', version: '4.0.42.Final'
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.5'
compile group: 'org.apache.maven', name: 'maven-core', version: '3.0.5'
@@ -75,18 +74,7 @@ dependencies {
compile project(':common')
compile project(':amaterasu-sdk')
-//runtime dependency for spark
-provided('org.apache.spark:spark-repl_2.11:2.2.1')
-provided('org.apache.spark:spark-core_2.11:2.2.1')
-
-testCompile project(':common')
-testCompile "gradle.plugin.com.github.maiflai:gradle-scalatest:0.14"
-testRuntime 'org.pegdown:pegdown:1.1.0'
-testCompile 'junit:junit:4.11'
-testCompile 'org.scalatest:scalatest_2.11:3.0.2'
-testCompile 'org.scala-lang:scala-library:2.11.8'
-testCompile('org.apache.spark:spark-repl_2.11:2.2.1')
-testCompile group: 'org.slf4j', name: 'slf4j-api', version: '1.7.9'
+
}
diff --git
a/executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/MesosActionsExecutor.scala
b/executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/MesosActionsExecutor.scala
index 9ab75be..90c2001 100755
---
a/executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/MesosActionsExecutor.scala
+++
b/executor/src/main/scala/org/apache/amaterasu/executor/mesos/executors/MesosActionsExecutor.scala
@@ -26,7 +26,6 @@ import
org.apache.amaterasu.executor.common.executors.ProvidersFactory
import org.apache.mesos.Protos._
import org.apache.mesos.protobuf.ByteString
import org.apache.mesos.{Executor, ExecutorDriver, MesosExecutorDriver}
-import org.apache.spark.SparkContext
import scala.collection.JavaConverters._
import scala.concurrent.ExecutionContext.Implicits.global
@@ -37,7 +36,6 @@ class MesosActionsExecutor extends Executor with Logging {
var master: String = _
var executorDriver: ExecutorDriver = _
- var sc: SparkContext = _
var jobId: String = _
var actionName: String = _
// var sparkScalaRunner: SparkScalaRunner = _
@@ -83,7 +81,7 @@ class MesosActionsExecutor extends Executor with Logging {
notifier = new MesosNotifier(driver)
notifier.info(s"Executor ${executorInfo.getExecutorId.getValue}
registered")
val outStream = new ByteArrayOutputStream()
-providersFactory = ProvidersFactory(data, jobId, outStream, notifier,
executorInfo.getExecutorId.getValue, hostName, propFile =
"./amaterasu.properties")
+providersFactory =
