[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2849 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-112894251 That's fine, but in the name of trying to clean up stale PRs, would you mind closing this PR? it's not mergeable and seems corrupted anyway. You can reopen another PR if you really want to. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user olegz commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-112884718 @srowen Iâd suggest to move this discussion to JIRA and see if we get a disposition there on the overall proposal and idea? The PR may not be in the mergable state and while GitHub is appropriate medium to discuss any technical issues, current discussion seems to go beyond that, hence my suggestion on moving it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user maidh91 commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-112638942 I really hope that this patch will become official part in Spark. I think @srowen is right that we should clean all messy things and open again later. Spark 1.4.0 is just released and Spark-Submit 2015 is happened today and they introduce many new features. It is pretty pity that this patch is not one of them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-112596449 @maidh91 this is not going to be merged. I'm suggesting it be closed actually. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user maidh91 commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-112594662 Yes, please fix existing conflicts and merge it. It will be perfect if you can merge with the latest version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-112561134 Although I'm pretty sure that's the resolution, sure, leave it open if you like. But this PR can't be merged and seems to have gotten messed up somehow; I'm narrowly asking you to not leave it both in that state and open. Close it, or resolve the conflicts / merge history stuff. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user olegz commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-112559171 Sean I am not sure I understand the "rejected" part, since no rejection has been issued (-1) in the JIRA. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-112557647 The update was just made by automated tools, not any person. As far as I can tell the proposal in the JIRA is rejected. The problem with this PR is that not only does it not merge but for some reason it has a lot of other commits in it and touches 820 files. Maybe a full rebase would fix it, not sure. But it can be closed in any event; it will stay here for posterity anyway. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user olegz commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-112551214 Well, I am waiting in the resolution of https://issues.apache.org/jira/browse/SPARK-3561 since it has recently been updated to "In Progress". I would rather update the PR, to make it mergable unless there is a different proposed approach which I would like to read about. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-112550321 Can you close this PR? it's no longer mergeable and looks borked at this point. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-100686174 > when will Spark finish to verify it? @maidh91 - Please follow the discussion on the JIRA issue to get this kind of information: [SPARK-3561](https://issues.apache.org/jira/browse/SPARK-3561) As for whether this patch still works, it hasn't been updated in a while and currently has a merge conflict, so probably not. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user maidh91 commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-97692131 Is this patch still working? when will Spark finish to verify it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-96770156 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-63709032 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user olegz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2849#discussion_r19648083
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1639,12 +1615,38 @@ object SparkContext extends Logging {
scheduler.initialize(backend)
(backend, scheduler)
+ case EXECUTION_CONTEXT(sparkUrl) =>
--- End diff --
That is correct. As you can see it is not being used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user olegz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2849#discussion_r19648060
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1639,12 +1615,38 @@ object SparkContext extends Logging {
scheduler.initialize(backend)
(backend, scheduler)
+ case EXECUTION_CONTEXT(sparkUrl) =>
+logInfo("Will use custom job execution context " + sparkUrl)
+sc.executionContext = Class.forName(sparkUrl).newInstance().
+asInstanceOf[JobExecutionContext]
+val scheduler = new NoOpTaskScheduler(sc)
+val backend = new LocalBackend(scheduler, 1)
+(backend, scheduler)
+
case _ =>
throw new SparkException("Could not parse Master URL: '" + master
+ "'")
}
}
}
-
+/**
+ * No-op implementation of TaskScheduler which is used in cases where
+ * execution of Spark DAG is delegate to an external execution environment,
+ * thus not relying on DAGScheduler nor TaskScheduler
+ */
+private class NoOpTaskScheduler(sc: SparkContext) extends
TaskSchedulerImpl(sc, 1) {
+ override val schedulingMode: SchedulingMode.SchedulingMode =
SchedulingMode.NONE
+ override def start(): Unit = {}
+ override def stop(): Unit = {}
+ override def submitTasks(taskSet: TaskSet): Unit = {}
+ override def cancelTasks(stageId: Int, interruptThread: Boolean) = {}
+ override def setDAGScheduler(dagScheduler: DAGScheduler): Unit = {}
+ override def defaultParallelism(): Int = 1
+ override def executorHeartbeatReceived(execId: String,
+ taskMetrics: Array[(Long, TaskMetrics)],
+blockManagerId: BlockManagerId): Boolean = true
+ override def applicationId(): String = sc.appName
--- End diff --
Thanks, I'll address it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/2849#discussion_r19647973
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1639,12 +1615,38 @@ object SparkContext extends Logging {
scheduler.initialize(backend)
(backend, scheduler)
+ case EXECUTION_CONTEXT(sparkUrl) =>
--- End diff --
If we use custom execution engine, DAGScheduler is not needed to initialize
right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/2849#discussion_r19647952
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1639,12 +1615,38 @@ object SparkContext extends Logging {
scheduler.initialize(backend)
(backend, scheduler)
+ case EXECUTION_CONTEXT(sparkUrl) =>
+logInfo("Will use custom job execution context " + sparkUrl)
+sc.executionContext = Class.forName(sparkUrl).newInstance().
+asInstanceOf[JobExecutionContext]
+val scheduler = new NoOpTaskScheduler(sc)
+val backend = new LocalBackend(scheduler, 1)
+(backend, scheduler)
+
case _ =>
throw new SparkException("Could not parse Master URL: '" + master
+ "'")
}
}
}
-
+/**
+ * No-op implementation of TaskScheduler which is used in cases where
+ * execution of Spark DAG is delegate to an external execution environment,
+ * thus not relying on DAGScheduler nor TaskScheduler
+ */
+private class NoOpTaskScheduler(sc: SparkContext) extends
TaskSchedulerImpl(sc, 1) {
+ override val schedulingMode: SchedulingMode.SchedulingMode =
SchedulingMode.NONE
+ override def start(): Unit = {}
+ override def stop(): Unit = {}
+ override def submitTasks(taskSet: TaskSet): Unit = {}
+ override def cancelTasks(stageId: Int, interruptThread: Boolean) = {}
+ override def setDAGScheduler(dagScheduler: DAGScheduler): Unit = {}
+ override def defaultParallelism(): Int = 1
+ override def executorHeartbeatReceived(execId: String,
+ taskMetrics: Array[(Long, TaskMetrics)],
+blockManagerId: BlockManagerId): Boolean = true
+ override def applicationId(): String = sc.appName
--- End diff --
Please don't use `appName` for Application ID because Application ID should
be unique.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-60014038 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3561] Initial commit to provide pluggab...
Github user olegz commented on the pull request: https://github.com/apache/spark/pull/2849#issuecomment-59807570 @andrewor14 done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
