Github user cleaton commented on a diff in the pull request:
https://github.com/apache/spark/pull/4338#discussion_r24038573
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
---
@@ -205,6 +205,33 @@ class StreamingContextSuite extends
Github user cleaton closed the pull request at:
https://github.com/apache/spark/pull/3868
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user cleaton commented on the pull request:
https://github.com/apache/spark/pull/4338#issuecomment-72718673
@tdas I will address it now. I have one question though. Don't you think
there at least should be some sort of timeout while waiting for the receivers
to deregist
Github user cleaton commented on the pull request:
https://github.com/apache/spark/pull/3868#issuecomment-72662664
@tdas I have created a new master branch PR. You can find it here:
https://github.com/apache/spark/pull/4338
---
If your project is set up for it, you can reply to this
GitHub user cleaton opened a pull request:
https://github.com/apache/spark/pull/4338
[STREAMING] SPARK-4986 Wait for receivers to deregister and receiver job to
terminate
A slow receiver might not have enough time to shutdown cleanly even when
graceful shutdown is used. This PR
Github user cleaton commented on the pull request:
https://github.com/apache/spark/pull/3868#issuecomment-72555661
I will rebase to master tomorrow and submit it as a new PR for SPARK-4986.
There is still the problem of how to handle the gracefulStopTimeout option, any
thoughts about
Github user cleaton commented on the pull request:
https://github.com/apache/spark/pull/3868#issuecomment-72164837
@tdas I have a new branch here:
https://github.com/cleaton/spark/compare/apache:branch-1.2...receiverstop
Still missing code to handle gracefulStopTimeout but
Github user cleaton commented on the pull request:
https://github.com/apache/spark/pull/3868#issuecomment-72158188
Thanks @tdas
I'll create a new minimal PR with only the receiver shutdown part and the
unit test. The main problem is how to deal with the timeout then. Th
Github user cleaton commented on the pull request:
https://github.com/apache/spark/pull/3868#issuecomment-71982990
Thank you for the comments @tdas . I think the diff file for this PR has
become a bit misleading. It looks like there is more change than there actually
is. This is a
Github user cleaton commented on the pull request:
https://github.com/apache/spark/pull/3868#issuecomment-71843209
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user cleaton commented on the pull request:
https://github.com/apache/spark/pull/3868#issuecomment-71375392
@tdas I've added a unit test now covering slow receiver shutdown. What do
you think about the approach now? Thanks
---
If your project is set up for it, you can
Github user cleaton commented on the pull request:
https://github.com/apache/spark/pull/3868#issuecomment-68707568
OK sounds great. :+1:
I can prepare an implementation of (2). Bit busy now, but I think I can
have something to review in a week.
Any specific unit test
Github user cleaton commented on the pull request:
https://github.com/apache/spark/pull/3868#issuecomment-68678753
@tdas Thank you for the input.
Yes, the main purpose of this patch is to make ReceiverTracker graceful by
waiting for ssc.sparkContext.runJob(tempRDD
Github user cleaton commented on the pull request:
https://github.com/apache/spark/pull/3868#issuecomment-68657548
@JoshRosen Thank you for all the comments. I will update accordingly :).
Regarding the waitUntilDone function, yes I assume there must be something
like this in
Github user cleaton commented on the pull request:
https://github.com/apache/spark/pull/3868#issuecomment-68656608
@tdas yes I think I can, I will take a look at it. Thanks for the feedback.
---
If your project is set up for it, you can reply to this email and have your
reply appear
15 matches
Mail list logo
