Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-72959021
Agreed. I will file a jira. We should discuss the issue there. This one was
more of a break-fix, but that would be a more elaborate fix.
---
If your project is s
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3655
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-72949130
I can merge this for now and we can focus on that issue later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as wel
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-72789875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-72789869
[Test build #26710 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26710/consoleFull)
for PR 3655 at commit
[`5e2e7ad`](https://gith
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-72784728
@harishreedharan This begs a higher level questions of whether the write
ahead log (which is the probably component to fail) should have its own retries
independent of the r
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-72784691
[Test build #26710 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26710/consoleFull)
for PR 3655 at commit
[`5e2e7ad`](https://githu
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-72784543
ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enable
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-70015327
No, this does prevent data loss - basically if the store fails multiple
times, we shutdown the receiver completely. So the new receiver which gets
started starts
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-69684946
So I think the aim of this patch is to fix the recoverable problems of data
store with retries, not prevent data loss. That's my thought, sorry for my
misunderstanding.
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-69684611
Hi @harishreedharan , After carefully looking at the code, I think data
will not be lost even in such failure situation. For example, if we meet
exception in `onPushBlo
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-69384366
@tdas Any comments on this one, or is this one ready to go in?
---
If your project is set up for it, you can reply to this email and have your
reply appear on Git
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/3655#discussion_r21766800
--- Diff:
external/kafka/src/main/scala/org/apache/spark/streaming/kafka/ReliableKafkaReceiver.scala
---
@@ -201,12 +201,31 @@ class ReliableKafkaRec
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3655#discussion_r21586197
--- Diff:
external/kafka/src/main/scala/org/apache/spark/streaming/kafka/ReliableKafkaReceiver.scala
---
@@ -201,12 +201,31 @@ class ReliableKafkaReceiver[
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-66399109
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-66399105
[Test build #24282 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24282/consoleFull)
for PR 3655 at commit
[`5e2e7ad`](https://gith
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-66398284
Thanks Hari, seems this is a simple solution. BTW should we make `count =
3` as a configurable parameter? For others LGTM.
Original thoughts of introducing pen
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/3655#issuecomment-66393421
I messed up the jira number in the commit. Please fix it when merging.
---
If your project is set up for it, you can reply to this email and have your
reply appea
18 matches
Mail list logo