[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146729797 Why is this needed? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146730487 [Test build #43452 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43452/consoleFull) for PR 9041 at commit

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146730942 LGTM. I'd rephrase the description to say "up to a minute" since I was a little scared when reading the original one. "OMG all my apps are going to wait a minute before

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/9041#discussion_r41589229 --- Diff: external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePollingInputDStream.scala --- @@ -93,7 +93,11 @@ private[streaming] class

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread harishreedharan
GitHub user harishreedharan opened a pull request: https://github.com/apache/spark/pull/9041 [SPARK-11019][streaming][flume] Gracefully shutdown Flume receiver th… …reads. Wait for a minute for the receiver threads to shutdown before interrupting them. You can merge

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146734813 [Test build #43452 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43452/console) for PR 9041 at commit

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146723009 [Test build #43444 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43444/consoleFull) for PR 9041 at commit

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146725895 [Test build #43444 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43444/console) for PR 9041 at commit

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146725914 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146730016 This basically makes testing a bit more predictable. Sometimes, we end up hitting a situation where the last transaction is still not completed and the

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146735090 LGTM. Merging it master and branch 1.5 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146734952 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146734953 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146735295 Actually, there were small conflicts in branch 1.5, so I only merged to master. Seems fine, not a highpriority bug that 1.5 should get it. --- If your project is set up

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9041 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146724990 /cc @tdas --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146730211 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146732540 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146732559 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146736757 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146736758 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146722776 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146722754 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146730242 Gotcha. How long generally does it take to shutdown? I mean, how long typically will the awaitTermination wait? --- If your project is set up for it, you can reply to

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146730398 In most cases, less than 5-10 seconds - just for the last in-flight batch to be done. This is not going to slow our unit test runs down. I mostly hit it in our

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146730368 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146730355 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146733002 [Test build #43454 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43454/consoleFull) for PR 9041 at commit

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146736693 [Test build #43454 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43454/console) for PR 9041 at commit

[GitHub] spark pull request: [SPARK-11019][streaming][flume] Gracefully shu...

2015-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9041#issuecomment-146725915 Test FAILed. Refer to this link for build results (access rights to CI server needed):