[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
GitHub user uncleGen opened a pull request: https://github.com/apache/spark/pull/4008 [SPARK-5205][Streaming]:Inconsistent behaviour between Streaming job and others, when click kill link in WebUI The "kill" link is used to kill a stage in job. It works in any kinds of Spark job but Spark Streaming. To be specific, we can only kill the stage which is used to run "Receiver", but not kill the "Receivers". Well, the stage can be killed and cleaned from the ui, but the receivers are still alive and receiving data. I think it dose not fit with the common sense. IMHO, killing the "receiver" stage means kill the "receivers" and stopping receiving data. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uncleGen/spark master-clean-150112 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4008.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4008 commit 9c77f403084c9971a82ae98d5f28704fa62c5269 Author: uncleGen Date: 2015-01-12T15:57:57Z kill streaming receiver stage through ui commit cf0342f987282679cc65ef4765658e0f31d929c9 Author: uncleGen Date: 2015-01-12T16:01:54Z fix commit 5f22ea4cbc15660a2444e566bde0f34ad69e Author: uncleGen Date: 2015-01-12T16:17:51Z description --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69601254 [Test build #25422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25422/consoleFull) for PR 4008 at commit [`5f22ea4`](https://github.com/apache/spark/commit/5f22ea4cbc15660a2444e566bde0f34ad69e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69601423 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25422/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69601417 [Test build #25422 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25422/consoleFull) for PR 4008 at commit [`5f22ea4`](https://github.com/apache/spark/commit/5f22ea4cbc15660a2444e566bde0f34ad69e). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69602242 [Test build #25423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25423/consoleFull) for PR 4008 at commit [`0ba5d05`](https://github.com/apache/spark/commit/0ba5d05a13dcc1b32888cd6d4b93feff5d936ac1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69614205 [Test build #25423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25423/consoleFull) for PR 4008 at commit [`0ba5d05`](https://github.com/apache/spark/commit/0ba5d05a13dcc1b32888cd6d4b93feff5d936ac1). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69614212 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25423/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69645179 @JoshRosen Can you confirm what is the current behavior of running tasks when a stage is cancelled? Are the tasks cancelled? What are existing hooks? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/4008#issuecomment-69657578
To quickly trace through the current behavior:
- The ["kill" link
handler](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/jobs/StagesTab.scala#L42)
calls `SparkContext.cancelStage()`.
-
[This](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L1436),
in turn, calls DAGScheduler.cancelStage().
- In the
[handleStageCancellation](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1148),
handler, DAGScheduler finds all jobs that depend on a stage and cancels those
jobs.
-
[This](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1165),
in turn,
[fails](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1194),
the job and all stages used only by that job ("independent stages"). When
failing a stage, DAGScheduler [tries to
cancel](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1221)
any tasks running in that stage. For a while, the Mesos scheduler backend
didn't support task cancellation, but this has been fixed. Task cancellation
is asynchronous and best-effort, but the expected behavior is that canceling a
stage will kill any running tasks in that stage.
I guess the issue here is that the task cancellation isn't sufficient to
clean up the receiver because it's run in a separate thread that's not cleaned
up when the task exits. We can't necessarily rely on threads being interrupted
when they're cancelled (see
https://github.com/apache/spark/commit/432201c7ee9e1ea1d70a6418cbad1c5ad2653ed3),
so this seems like a reasonable approach.
I have a couple of code-level comments about the approach proposed here,
though, so let me comment on those in-line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69658255 Oh, and to provide a little more context, the reason that we're able to cancel most tasks even though we don't perform thread interrupts is that we have an [InterruptibleIterator](https://github.com/uncleGen/spark/blob/master-clean-150112/core/src/main/scala/org/apache/spark/InterruptibleIterator.scala) class that causes canceled tasks to throw exceptions when they try to consume their next element of input. That doesn't seem to take effect here, though, because our receiver task doesn't consume any input. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4008#discussion_r22828138
--- Diff: core/src/main/java/org/apache/spark/TaskContext.java ---
@@ -94,6 +94,14 @@ static void unset() {
@Deprecated
public abstract void addOnCompleteCallback(final Function0 f);
+ /**
+ * Add a callback function to be executed on task stop.
+ * This will be called when task is being killed.
+ *
+ * @param f Callback function.
+ */
+ public abstract void addOnStopCallback(final Function1 f);
--- End diff --
I think this name is potentially confusing, since users might get confused
about the difference between "stop" and "completion". I'd prefer something
like "onKill" to make it more obvious that this is only called for killed tasks.
Also, what is the string argument that's passed in to the callback?
Also, this seems similar to the deprecated `addOnCompleteCallback`. Maybe
we should fold the new callback into `TaskCompletionListener` instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4008#discussion_r22828514
--- Diff: core/src/main/java/org/apache/spark/TaskContext.java ---
@@ -94,6 +94,14 @@ static void unset() {
@Deprecated
public abstract void addOnCompleteCallback(final Function0 f);
+ /**
+ * Add a callback function to be executed on task stop.
+ * This will be called when task is being killed.
+ *
+ * @param f Callback function.
+ */
+ public abstract void addOnStopCallback(final Function1 f);
--- End diff --
Unfortunately, I guess we can't add a new method to
`TaskCompletionListener` because it's a public trait that's designed to be
extended from Java, so adding any new methods will break source-compatibility
for those users:
https://github.com/uncleGen/spark/blob/master-clean-150112/core/src/main/scala/org/apache/spark/util/TaskCompletionListener.scala
I suppose we could add a separate TaskKillListener or something like that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69680034 @JoshRosen Thanks for your patience, I will optimize my code based on your comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69701688 [Test build #25460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25460/consoleFull) for PR 4008 at commit [`2349565`](https://github.com/apache/spark/commit/23495651c7efea342d68602070d62f8065b140cc). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69701818 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25460/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69701816 [Test build #25460 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25460/consoleFull) for PR 4008 at commit [`2349565`](https://github.com/apache/spark/commit/23495651c7efea342d68602070d62f8065b140cc). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait TaskKilledListener extends EventListener ` * `class TaskKilledListenerException(errorMessages: Seq[String]) extends Exception ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69702356 [Test build #25461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25461/consoleFull) for PR 4008 at commit [`81d91eb`](https://github.com/apache/spark/commit/81d91ebdd5151cc6e938631d0bf853711e3c61f7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69707664 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25461/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69707660 [Test build #25461 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25461/consoleFull) for PR 4008 at commit [`81d91eb`](https://github.com/apache/spark/commit/81d91ebdd5151cc6e938631d0bf853711e3c61f7). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait TaskKilledListener extends EventListener ` * `class TaskKilledListenerException(errorMessages: Seq[String]) extends Exception ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69710506 [Test build #25464 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25464/consoleFull) for PR 4008 at commit [`12a9b4e`](https://github.com/apache/spark/commit/12a9b4eca7e161d5e91f3e35e232e0459cdc5077). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69717935 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25464/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69717923 [Test build #25464 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25464/consoleFull) for PR 4008 at commit [`12a9b4e`](https://github.com/apache/spark/commit/12a9b4eca7e161d5e91f3e35e232e0459cdc5077). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait TaskKilledListener extends EventListener ` * `class TaskKilledListenerException(errorMessages: Seq[String]) extends Exception ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen commented on the pull request:
https://github.com/apache/spark/pull/4008#issuecomment-69719446
@JoshRosen how to fix these two errors? According to the hint, I need to
add
"ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.TaskContext.addTaskKilledListener")",
but still failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69860462 [Test build #25500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25500/consoleFull) for PR 4008 at commit [`61cda0d`](https://github.com/apache/spark/commit/61cda0d732f81a9a9aa1238aaa538570e2b96c10). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69864877 @uncleGen I think you need to add the excludes to the 1.3 section, not 1.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69865066 [Test build #25500 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25500/consoleFull) for PR 4008 at commit [`61cda0d`](https://github.com/apache/spark/commit/61cda0d732f81a9a9aa1238aaa538570e2b96c10). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait TaskKilledListener extends EventListener ` * `class TaskKilledListenerException(errorMessages: Seq[String]) extends Exception ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69865071 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25500/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69865814 [Test build #25506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25506/consoleFull) for PR 4008 at commit [`2e454c3`](https://github.com/apache/spark/commit/2e454c39e4f94bc9d9db224602c156958ee4b70d). * This patch **does not merge cleanly**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69871554 [Test build #25506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25506/consoleFull) for PR 4008 at commit [`2e454c3`](https://github.com/apache/spark/commit/2e454c39e4f94bc9d9db224602c156958ee4b70d). * This patch **passes all tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_: * `trait TaskKilledListener extends EventListener ` * `class TaskKilledListenerException(errorMessages: Seq[String]) extends Exception ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69871556 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25506/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4008#discussion_r23064794
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala
---
@@ -23,10 +23,20 @@ import scala.language.existentials
import akka.actor._
-import org.apache.spark.{Logging, SerializableWritable, SparkEnv,
SparkException}
+import org.apache.spark._
import org.apache.spark.SparkContext._
import org.apache.spark.streaming.{StreamingContext, Time}
import org.apache.spark.streaming.receiver.{Receiver,
ReceiverSupervisorImpl, StopReceiver}
+import org.apache.spark.streaming.scheduler.ReceivedBlockInfo
--- End diff --
It is Intellij IDEA's automatic completion work, sorry for my carelessness.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4008#discussion_r23064804
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala
---
@@ -274,6 +284,7 @@ class ReceiverTracker(ssc: StreamingContext,
skipReceiverLaunch: Boolean = false
val receiver = iterator.next()
val supervisor = new ReceiverSupervisorImpl(
receiver, SparkEnv.get, serializableHadoopConf.value,
checkpointDirOption)
+context.addTaskKilledCallback(context => supervisor.stop("Receiver
was killed.", None))
--- End diff --
Make sense
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-70213336 [Test build #25635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25635/consoleFull) for PR 4008 at commit [`a87e063`](https://github.com/apache/spark/commit/a87e063917d482a58cba7e118c038e24126b10db). * This patch **does not merge cleanly**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-70213415 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25635/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-70213414 [Test build #25635 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25635/consoleFull) for PR 4008 at commit [`a87e063`](https://github.com/apache/spark/commit/a87e063917d482a58cba7e118c038e24126b10db). * This patch **fails Scala style tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_: * `trait TaskKilledListener extends EventListener ` * `class TaskKilledListenerException(errorMessages: Seq[String]) extends Exception ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-70218720 [Test build #25642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25642/consoleFull) for PR 4008 at commit [`d202e6e`](https://github.com/apache/spark/commit/d202e6e1a8970c3e1f6660a03bc5774c2e40db54). * This patch **does not merge cleanly**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-70227990 [Test build #25642 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25642/consoleFull) for PR 4008 at commit [`d202e6e`](https://github.com/apache/spark/commit/d202e6e1a8970c3e1f6660a03bc5774c2e40db54). * This patch **passes all tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_: * `trait TaskKilledListener extends EventListener ` * `class TaskKilledListenerException(errorMessages: Seq[String]) extends Exception ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-70227997 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25642/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
GitHub user uncleGen opened a pull request: https://github.com/apache/spark/pull/4135 [SPARK-5205][Streaming]:Inconsistent behaviour between Streaming job and others, when click kill link in WebUI The "kill" link is used to kill a stage in job. It works in any kinds of Spark job but Spark Streaming. To be specific, we can only kill the stage which is used to run "Receiver", but not kill the "Receivers". Well, the stage can be killed and cleaned from the ui, but the receivers are still alive and receiving data. I think it dose not fit with the common sense. IMHO, killing the "receiver" stage means kill the "receivers" and stopping receiving data. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uncleGen/spark master-clean-150121 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4135.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4135 commit c90a288ca6fde01008ff3ab5d04970c8f120c4b1 Author: uncleGen Date: 2015-01-21T08:58:10Z BUG FIX: Inconsistent behaviour between Streaming job and others, when click kill link in WebUI --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-70804703 @JoshRosen Since my local branch is out of date and this PR contains merge conflicts, I will open a [new PR](https://github.com/apache/spark/pull/4135) and close this first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen closed the pull request at: https://github.com/apache/spark/pull/4008 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70805185 [Test build #25882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25882/consoleFull) for PR 4135 at commit [`c90a288`](https://github.com/apache/spark/commit/c90a288ca6fde01008ff3ab5d04970c8f120c4b1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70805248 [Test build #25882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25882/consoleFull) for PR 4135 at commit [`c90a288`](https://github.com/apache/spark/commit/c90a288ca6fde01008ff3ab5d04970c8f120c4b1). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait TaskKilledListener extends EventListener ` * `class TaskKilledListenerException(errorMessages: Seq[String]) extends Exception ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70805253 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25882/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70806362 [Test build #25884 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25884/consoleFull) for PR 4135 at commit [`fc3be2c`](https://github.com/apache/spark/commit/fc3be2c0d3cd3e8f2f15a4a4f0fa504b28164d38). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70806543 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25884/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70806538 [Test build #25884 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25884/consoleFull) for PR 4135 at commit [`fc3be2c`](https://github.com/apache/spark/commit/fc3be2c0d3cd3e8f2f15a4a4f0fa504b28164d38). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70807559 [Test build #25886 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25886/consoleFull) for PR 4135 at commit [`d85dede`](https://github.com/apache/spark/commit/d85dede41dce4a32748037871241da1620ada564). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70816298 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25886/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-70816290 [Test build #25886 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25886/consoleFull) for PR 4135 at commit [`d85dede`](https://github.com/apache/spark/commit/d85dede41dce4a32748037871241da1620ada564). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait TaskKilledListener extends EventListener ` * `class TaskKilledListenerException(errorMessages: Seq[String]) extends Exception ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-87870087 Any update on this? This patch looked good to me overall; the main thrust of my feedback was that the tests here didn't seem sufficient to actually exercise the code added in this patch, but I think that shouldn't be too hard to fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-87908164 @JoshRosen Sorry for my laziness. I will update it as soon as possible --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-87918010 @uncleGen No worries; I'm not in a huge rush, but just wanted to check in to see if you planned to update this. Sorry that the review process has been so slow; thanks for your patience! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen closed the pull request at: https://github.com/apache/spark/pull/4135 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
GitHub user uncleGen reopened a pull request: https://github.com/apache/spark/pull/4135 [SPARK-5205][Streaming]:Inconsistent behaviour between Streaming job and others, when click kill link in WebUI The "kill" link is used to kill a stage in job. It works in any kinds of Spark job but Spark Streaming. To be specific, we can only kill the stage which is used to run "Receiver", but not kill the "Receivers". Well, the stage can be killed and cleaned from the ui, but the receivers are still alive and receiving data. I think it dose not fit with the common sense. IMHO, killing the "receiver" stage means kill the "receivers" and stopping receiving data. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uncleGen/spark master-clean-150121 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4135.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4135 commit c90a288ca6fde01008ff3ab5d04970c8f120c4b1 Author: uncleGen Date: 2015-01-21T08:58:10Z BUG FIX: Inconsistent behaviour between Streaming job and others, when click kill link in WebUI commit fc3be2c0d3cd3e8f2f15a4a4f0fa504b28164d38 Author: uncleGen Date: 2015-01-21T09:16:12Z fix commit d85dede41dce4a32748037871241da1620ada564 Author: uncleGen Date: 2015-01-21T09:23:59Z style fix commit f3162618d3eb1de3e6690e1403bf1943145d316a Author: uncleGen Date: 2015-02-03T02:36:52Z resolve merge conflicts commit 168115cb364285867bc9e7af400d48922712c71c Author: uncleGen Date: 2015-03-02T03:46:42Z resolve merge conflicts commit 047931c531b7d307de201ba74ed5dcb7da7c6559 Author: uncleGen Date: 2015-03-02T05:24:18Z resolve merge conflicts commit d8b57dfff760ce7a3b2413c3dcada3e1572fb7e8 Author: uncleGen Date: 2015-03-09T03:28:37Z Merge branch 'master-clean' into master-clean-150121 commit 4417ff06aeeea63b1a2f4e8a12d736a050abf057 Author: uncleGen Date: 2015-03-09T08:19:02Z update commit 963556d4b64c81e56913a2860f596c6b25b1286d Author: uncleGen Date: 2015-03-10T03:14:15Z add unit test commit f997698a0233d351941fd7ddd9577a659806281e Author: uncleGen Date: 2015-03-10T03:42:03Z fix unit test commit 705118453ab6f9869ac090ca2bac009f167869cd Author: uncleGen Date: 2015-03-10T08:57:21Z minor fix commit 92fb864e384e298eead53f10844365ea1887a929 Author: uncleGen Date: 2015-03-12T11:41:33Z roll back commit 98166e7e8b43d1a912b2b6a468eb6d8f1f0297f2 Author: uncleGen Date: 2015-03-12T11:44:41Z minor fix commit baa175898df8d36b36ec9b9494ebbd117acfae87 Author: uncleGen Date: 2015-03-12T11:47:23Z minor fix commit fe3e5d52bf8680a021b00dd598cdc5a3f1c7df7e Author: uncleGen Date: 2015-03-12T11:51:11Z minor fix commit fb9716d3feee4523a3ee4cddd0a9c0926e228099 Author: uncleGen Date: 2015-03-12T11:52:33Z minor fix commit 02bf9a936f798b7eb40dcd1183ea14b5c2125deb Author: uncleGen Date: 2015-03-12T12:17:46Z minor fix commit 2bac6564a0a1153ec1366bfa4da085a6e89d07d5 Author: uncleGen Date: 2015-03-12T13:49:21Z minor fix commit 2544f8e5ad0e205e4043bd47646f57d5ed1f42a8 Author: uncleGen Date: 2015-03-17T13:52:36Z roll back to original approach commit b5237d86ddfcfd0b1b9b5bcd6952902ee1d843b6 Author: uncleGen Date: 2015-03-17T14:03:10Z resolve merge conflict commit 96f21820250bec9c7286489baa0f7dbd96758df6 Author: uncleGen Date: 2015-03-18T02:38:58Z resolve scala style error commit b2916fb7cdb6047eb618c2da767d18eb5a4e0caf Author: uncleGen Date: 2015-03-18T09:20:55Z minor fix commit 77df65a1d96bb98ee7617590825d63c8ee6a26f9 Author: uncleGen Date: 2015-03-31T06:30:53Z Merge branch 'master-clean-tmp' into master-clean-150121 Conflicts: streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala streaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala commit e6724ecac594e818973a6d813c4d012fca1bd06d Author: uncleGen Date: 2015-03-31T06:40:01Z minor fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-87964217 [Test build #29461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29461/consoleFull) for PR 4135 at commit [`e6724ec`](https://github.com/apache/spark/commit/e6724ecac594e818973a6d813c4d012fca1bd06d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-87967179 [Test build #29461 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29461/consoleFull) for PR 4135 at commit [`e6724ec`](https://github.com/apache/spark/commit/e6724ecac594e818973a6d813c4d012fca1bd06d). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait TaskInterruptionListener extends EventListener ` * `class TaskInterruptionListenerException(errorMessages: Seq[String]) extends Exception ` * `class UDFRegistration(object):` * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-87967183 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29461/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-87967820 @JoshRosen Your comments are reasonable, and I have improved related code just as what you pointed. For the test suite, I just check if the state of `Receiver` and `ReceiverSupervisor` are correct. In the previous way, killing `task` could not stop `Receiver` and `ReceiverSupervisor` properly. I think it is enough. @tdas, what is your opinion? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen closed the pull request at: https://github.com/apache/spark/pull/4135 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
GitHub user uncleGen reopened a pull request: https://github.com/apache/spark/pull/4135 [SPARK-5205][Streaming]:Inconsistent behaviour between Streaming job and others, when click kill link in WebUI The "kill" link is used to kill a stage in job. It works in any kinds of Spark job but Spark Streaming. To be specific, we can only kill the stage which is used to run "Receiver", but not kill the "Receivers". Well, the stage can be killed and cleaned from the ui, but the receivers are still alive and receiving data. I think it dose not fit with the common sense. IMHO, killing the "receiver" stage means kill the "receivers" and stopping receiving data. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uncleGen/spark master-clean-150121 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4135.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4135 commit c90a288ca6fde01008ff3ab5d04970c8f120c4b1 Author: uncleGen Date: 2015-01-21T08:58:10Z BUG FIX: Inconsistent behaviour between Streaming job and others, when click kill link in WebUI commit fc3be2c0d3cd3e8f2f15a4a4f0fa504b28164d38 Author: uncleGen Date: 2015-01-21T09:16:12Z fix commit d85dede41dce4a32748037871241da1620ada564 Author: uncleGen Date: 2015-01-21T09:23:59Z style fix commit f3162618d3eb1de3e6690e1403bf1943145d316a Author: uncleGen Date: 2015-02-03T02:36:52Z resolve merge conflicts commit 168115cb364285867bc9e7af400d48922712c71c Author: uncleGen Date: 2015-03-02T03:46:42Z resolve merge conflicts commit 047931c531b7d307de201ba74ed5dcb7da7c6559 Author: uncleGen Date: 2015-03-02T05:24:18Z resolve merge conflicts commit d8b57dfff760ce7a3b2413c3dcada3e1572fb7e8 Author: uncleGen Date: 2015-03-09T03:28:37Z Merge branch 'master-clean' into master-clean-150121 commit 4417ff06aeeea63b1a2f4e8a12d736a050abf057 Author: uncleGen Date: 2015-03-09T08:19:02Z update commit 963556d4b64c81e56913a2860f596c6b25b1286d Author: uncleGen Date: 2015-03-10T03:14:15Z add unit test commit f997698a0233d351941fd7ddd9577a659806281e Author: uncleGen Date: 2015-03-10T03:42:03Z fix unit test commit 705118453ab6f9869ac090ca2bac009f167869cd Author: uncleGen Date: 2015-03-10T08:57:21Z minor fix commit 92fb864e384e298eead53f10844365ea1887a929 Author: uncleGen Date: 2015-03-12T11:41:33Z roll back commit 98166e7e8b43d1a912b2b6a468eb6d8f1f0297f2 Author: uncleGen Date: 2015-03-12T11:44:41Z minor fix commit baa175898df8d36b36ec9b9494ebbd117acfae87 Author: uncleGen Date: 2015-03-12T11:47:23Z minor fix commit fe3e5d52bf8680a021b00dd598cdc5a3f1c7df7e Author: uncleGen Date: 2015-03-12T11:51:11Z minor fix commit fb9716d3feee4523a3ee4cddd0a9c0926e228099 Author: uncleGen Date: 2015-03-12T11:52:33Z minor fix commit 02bf9a936f798b7eb40dcd1183ea14b5c2125deb Author: uncleGen Date: 2015-03-12T12:17:46Z minor fix commit 2bac6564a0a1153ec1366bfa4da085a6e89d07d5 Author: uncleGen Date: 2015-03-12T13:49:21Z minor fix commit 2544f8e5ad0e205e4043bd47646f57d5ed1f42a8 Author: uncleGen Date: 2015-03-17T13:52:36Z roll back to original approach commit b5237d86ddfcfd0b1b9b5bcd6952902ee1d843b6 Author: uncleGen Date: 2015-03-17T14:03:10Z resolve merge conflict commit 96f21820250bec9c7286489baa0f7dbd96758df6 Author: uncleGen Date: 2015-03-18T02:38:58Z resolve scala style error commit b2916fb7cdb6047eb618c2da767d18eb5a4e0caf Author: uncleGen Date: 2015-03-18T09:20:55Z minor fix commit 77df65a1d96bb98ee7617590825d63c8ee6a26f9 Author: uncleGen Date: 2015-03-31T06:30:53Z Merge branch 'master-clean-tmp' into master-clean-150121 Conflicts: streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala streaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala commit e6724ecac594e818973a6d813c4d012fca1bd06d Author: uncleGen Date: 2015-03-31T06:40:01Z minor fix commit e05e87f3a5f3f7325f7e31591d7556afdef8b8dd Author: uncleGen Date: 2015-03-31T06:56:56Z minor fix commit e26a207b1f53ce426677221a03fe100fd0a72bf4 Author: uncleGen Date: 2015-03-31T07:55:45Z update test suite --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-87987214 [Test build #29469 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29469/consoleFull) for PR 4135 at commit [`e26a207`](https://github.com/apache/spark/commit/e26a207b1f53ce426677221a03fe100fd0a72bf4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-87990173 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29469/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-87990165 [Test build #29469 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29469/consoleFull) for PR 4135 at commit [`e26a207`](https://github.com/apache/spark/commit/e26a207b1f53ce426677221a03fe100fd0a72bf4). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait TaskInterruptionListener extends EventListener ` * `class TaskInterruptionListenerException(errorMessages: Seq[String]) extends Exception ` * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-88013418 wait for a moment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-88064582 [Test build #29478 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29478/consoleFull) for PR 4135 at commit [`3a68742`](https://github.com/apache/spark/commit/3a68742a5631e9492ebdf29813ba547e1e4842bf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-88089542 [Test build #29478 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29478/consoleFull) for PR 4135 at commit [`3a68742`](https://github.com/apache/spark/commit/3a68742a5631e9492ebdf29813ba547e1e4842bf). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait TaskInterruptionListener extends EventListener ` * `class TaskInterruptionListenerException(errorMessages: Seq[String]) extends Exception ` * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-88089553 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29478/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-88342325 @JoshRosen Your comments are reasonable, and I have improved the relevant codes just what you pointed. And more, I add some unit tests about the behavior about `TaskInterruptionListener`. In the same time, I left some unit test in `ReceiverSuite` to test the behavior about `Receiver` and `ReceiverSupervisor`. Maybe, @tdas has some suggestions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-90405475 [Test build #29784 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29784/consoleFull) for PR 4135 at commit [`debfbac`](https://github.com/apache/spark/commit/debfbacd088d910351aa3fb0070e9ad3ea8ffce3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-90455405 [Test build #29784 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29784/consoleFull) for PR 4135 at commit [`debfbac`](https://github.com/apache/spark/commit/debfbacd088d910351aa3fb0070e9ad3ea8ffce3). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-90455427 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29784/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-90476533 Ha~llo~, @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4135#discussion_r28023348
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala ---
@@ -64,6 +64,53 @@ class TaskContextSuite extends FunSuite with
BeforeAndAfter with LocalSparkConte
verify(listener, times(1)).onTaskCompletion(any())
}
+ test("calls TaskInterruptionListener when task killed") {
+TaskContextSuite.interrupted = false
+sc = new SparkContext("local", "test")
+val rdd = new RDD[String](sc, List()) {
+ override def getPartitions = Array[Partition](StubPartition(0))
+ override def compute(split: Partition, context: TaskContext) = {
+context.addTaskInterruptionListener(context =>
TaskContextSuite.interrupted = true)
+Thread.sleep(1000)
+Iterator("test")
+ }
+}
+val closureSerializer = SparkEnv.get.closureSerializer.newInstance()
+val func = (c: TaskContext, i: Iterator[String]) => i.next()
+val task = new ResultTask[String, String](
+ 0, sc.broadcast(closureSerializer.serialize((rdd, func)).array),
rdd.partitions(0), Seq(), 0)
+
+task.kill(false)
+assert(TaskContextSuite.interrupted === false)
+
+val taskThread = new Thread("Task") {
+ override def run(): Unit = {
+task.run(0, 0)
+ }
+}
+taskThread.start()
+Thread.sleep(500)
--- End diff --
If possible, we should avoid adding any new Thread.sleep() calls in our
tests, since they slow things down and have been a major source of flakiness in
the past. How about adding a CountdownLatch to the TaskContexSuite companion
object and using that to block the main thread until the body of the task has
started executing and the interruption listener has been registered?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4135#discussion_r28024103
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala ---
@@ -295,6 +334,27 @@ class ReceiverSuite extends TestSuiteBase with
Timeouts with Serializable {
}
/**
+ * An implementation of Task running a NetworkReceiver used for testing
whether
+ * we can start/stop receiver properly.
+ */
+ class FakeReceiverTask(
+stageId: Int,
+executor: FakeReceiverSupervisor,
+prefLocs: Seq[TaskLocation] = Nil) extends Task[Unit](stageId, 0) {
--- End diff --
It looks like we never call this with a non-Nil value of `prefLocs`, so we
can probably drop the constructor argument and just hardcode `Nil` into the
`preferredLocations` method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4135#discussion_r28024057
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala ---
@@ -295,6 +334,27 @@ class ReceiverSuite extends TestSuiteBase with
Timeouts with Serializable {
}
/**
+ * An implementation of Task running a NetworkReceiver used for testing
whether
+ * we can start/stop receiver properly.
+ */
+ class FakeReceiverTask(
+stageId: Int,
--- End diff --
Style nit: these arguments should be indented another two spaces.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4135#discussion_r28023730
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala ---
@@ -129,6 +130,44 @@ class ReceiverSuite extends TestSuiteBase with
Timeouts with Serializable {
}
}
+ test("check the state of `Receiver` and `ReceiverSupervisor` when
receivers are killed") {
--- End diff --
I'd add `(SPARK-5205)` to the name of this test, if it will fit, or just
add a comment on the next line referencing the JIRA number.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4135#discussion_r28024130
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala ---
@@ -295,6 +334,27 @@ class ReceiverSuite extends TestSuiteBase with
Timeouts with Serializable {
}
/**
+ * An implementation of Task running a NetworkReceiver used for testing
whether
+ * we can start/stop receiver properly.
+ */
+ class FakeReceiverTask(
+stageId: Int,
+executor: FakeReceiverSupervisor,
--- End diff --
How about calling this `supervisor` instead of `executor`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4135#discussion_r28023642
--- Diff: project/MimaExcludes.scala ---
@@ -64,6 +64,10 @@ object MimaExcludes {
// SPARK-6492 Fix deadlock in SparkContext.stop()
ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.SparkContext.org$"
+
"apache$spark$SparkContext$$SPARK_CONTEXT_CONSTRUCTOR_LOCK")
+ ) ++ Seq(
--- End diff --
The indentation here looks a bit off. Mind indenting this back a bit so
that it lines up with the other closing braces (e.g. line 63)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-91069166 On closer inspection, my [previous comment](https://github.com/apache/spark/pull/4135#discussion_r26727249) about testing still stands > I notice that this test harness the logic from the real receiver tracker. Is this test suite actually covering the changes made to ReceiverTracker? In other words, if I were to go in and delete the callback registration code that you added to the real tracker, would that cause any tests to fail? If not, then I think we need to do additional work to test this, since the test isn't really testing anything if it can't expose bugs in the real code. > It seems like we'd like to test against a real ReceiverTracker using mock implementations of ReceiverSupervisor and Receiver. I'm not super-familiar with this corner of the streaming code, though, so maybe @tdas can chime in with suggestions on how to test this? Since the changes in this patch modify ReceiverTracker, it makes sense to test the actual ReceiverTracker while mocking other components. The updated test still doesn't exercise the real receiver tracker and thus isn't actually covering the streaming changes in this patch. I understand that properly testing ReceiverTracker might not be the easiest test to write, so perhaps this would require refactoring of ReceiverTracker to make the individual components more testable. I'll let @tdas chime in to decide on the importance of writing a real test for this. Sorry to be pedantic, but I'm a bit of a stickler for the importance of writing tests that actually exercise the code being modified, since I think it's very rare that it's truly impossible to test changes like this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4135#discussion_r28023099
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala ---
@@ -64,6 +64,53 @@ class TaskContextSuite extends FunSuite with
BeforeAndAfter with LocalSparkConte
verify(listener, times(1)).onTaskCompletion(any())
}
+ test("calls TaskInterruptionListener when task killed") {
+TaskContextSuite.interrupted = false
+sc = new SparkContext("local", "test")
+val rdd = new RDD[String](sc, List()) {
+ override def getPartitions = Array[Partition](StubPartition(0))
+ override def compute(split: Partition, context: TaskContext) = {
+context.addTaskInterruptionListener(context =>
TaskContextSuite.interrupted = true)
+Thread.sleep(1000)
+Iterator("test")
+ }
+}
+val closureSerializer = SparkEnv.get.closureSerializer.newInstance()
+val func = (c: TaskContext, i: Iterator[String]) => i.next()
+val task = new ResultTask[String, String](
+ 0, sc.broadcast(closureSerializer.serialize((rdd, func)).array),
rdd.partitions(0), Seq(), 0)
+
+task.kill(false)
--- End diff --
The task isn't running at this point, so what's the purpose of calling
`kill` here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4135#discussion_r28022727 --- Diff: core/src/main/scala/org/apache/spark/util/TaskInterruptionListener.scala --- @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.util + +import java.util.EventListener + +import org.apache.spark.TaskContext +import org.apache.spark.annotation.DeveloperApi + +/** + * :: DeveloperApi :: + * + * Listener providing a callback function to invoke when interrupt a task. --- End diff -- "when interrupt" -> "when interrupting" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4135#discussion_r28022977
--- Diff: core/src/main/scala/org/apache/spark/TaskContext.scala ---
@@ -106,6 +105,20 @@ abstract class TaskContext extends Serializable {
def addOnCompleteCallback(f: () => Unit)
/**
+ * Add a (Java friendly) listener to be executed on task interruption.
We add this
+ * listener for some more clean works. An example use is to stop
`receiver supervisor`
+ * properly.
+ */
+ def addTaskInterruptionListener(listener: TaskInterruptionListener):
TaskContext
+
+ /**
+ * Add a listener in the form of a Scala closure to be executed on task
interruption.
+ * We add this listener for some more clean works. An example use is to
stop `receiver
--- End diff --
I'd reword this slightly to be a bit more explicit:
> Add a listener in the form of a Scala closure to be executed on task
interruption. An example use is to cleanly shut down long-running receiver
tasks in Spark Streaming.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-76650327 @JoshRosen Could you please take a look again, thank you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen closed the pull request at: https://github.com/apache/spark/pull/4135 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
GitHub user uncleGen reopened a pull request: https://github.com/apache/spark/pull/4135 [SPARK-5205][Streaming]:Inconsistent behaviour between Streaming job and others, when click kill link in WebUI The "kill" link is used to kill a stage in job. It works in any kinds of Spark job but Spark Streaming. To be specific, we can only kill the stage which is used to run "Receiver", but not kill the "Receivers". Well, the stage can be killed and cleaned from the ui, but the receivers are still alive and receiving data. I think it dose not fit with the common sense. IMHO, killing the "receiver" stage means kill the "receivers" and stopping receiving data. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uncleGen/spark master-clean-150121 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4135.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4135 commit c90a288ca6fde01008ff3ab5d04970c8f120c4b1 Author: uncleGen Date: 2015-01-21T08:58:10Z BUG FIX: Inconsistent behaviour between Streaming job and others, when click kill link in WebUI commit fc3be2c0d3cd3e8f2f15a4a4f0fa504b28164d38 Author: uncleGen Date: 2015-01-21T09:16:12Z fix commit d85dede41dce4a32748037871241da1620ada564 Author: uncleGen Date: 2015-01-21T09:23:59Z style fix commit f3162618d3eb1de3e6690e1403bf1943145d316a Author: uncleGen Date: 2015-02-03T02:36:52Z resolve merge conflicts commit 168115cb364285867bc9e7af400d48922712c71c Author: uncleGen Date: 2015-03-02T03:46:42Z resolve merge conflicts commit 047931c531b7d307de201ba74ed5dcb7da7c6559 Author: uncleGen Date: 2015-03-02T05:24:18Z resolve merge conflicts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-76660466 [Test build #28156 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28156/consoleFull) for PR 4135 at commit [`047931c`](https://github.com/apache/spark/commit/047931c531b7d307de201ba74ed5dcb7da7c6559). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-76663454 @JoshRosen Could you please take a look again, thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-7004 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28156/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-76665998 [Test build #28156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28156/consoleFull) for PR 4135 at commit [`047931c`](https://github.com/apache/spark/commit/047931c531b7d307de201ba74ed5dcb7da7c6559). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait TaskKilledListener extends EventListener ` * `class TaskKilledListenerException(errorMessages: Seq[String]) extends Exception ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user uncleGen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-77514182 @pwendell Could you please take a look again, thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-4104 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-4344 [Test build #28374 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28374/consoleFull) for PR 4135 at commit [`047931c`](https://github.com/apache/spark/commit/047931c531b7d307de201ba74ed5dcb7da7c6559). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-4362 I think we discussed this on the first PR, but it would be good to add an explicit note here summarizing why we're not using `TaskCompletionListener`, since its documentation says that it's called on all task completion events, including success, failure, or cancellation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/4135#issuecomment-4567
Actually, this just occurred to me: why not use `addTaskCompletionListener`
and check `TaskContext.isInterrupted()` inside of your listener in order to
decide whether to perform the receiver shutdown? If this works, then this
patch becomes a two-line change to ReceiverTracker.scala:
```scala
context.addTaskCompletionListener { context =>
if (context.isInterrupted()) {
supervisor.stop("Receiver was killed.", None)
}
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-4769 Can you try using the existing `addTaskCompletionListener` with `context.isInterrupted()`, roll back the other changes / listener interfaces, and add a streaming unit test for this receiver cancellation behavior? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-8577 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28374/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-8576 [Test build #28374 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28374/consoleFull) for PR 4135 at commit [`047931c`](https://github.com/apache/spark/commit/047931c531b7d307de201ba74ed5dcb7da7c6559). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait TaskKilledListener extends EventListener ` * `class TaskKilledListenerException(errorMessages: Seq[String]) extends Exception ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-77989034 [Test build #28421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28421/consoleFull) for PR 4135 at commit [`963556d`](https://github.com/apache/spark/commit/963556d4b64c81e56913a2860f596c6b25b1286d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5205][Streaming]:Inconsistent behaviour...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4135#issuecomment-77991413 [Test build #28422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28422/consoleFull) for PR 4135 at commit [`f997698`](https://github.com/apache/spark/commit/f997698a0233d351941fd7ddd9577a659806281e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
