Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3832
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3832#issuecomment-68666982
There are few nits, but this LGTM even without it. I am merging this.
Thanks for this finding and solving this!
---
If your project is set up for it, you can reply to this
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3832#discussion_r22447244
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -255,6 +255,45 @@ class CheckpointSuite extends TestSuiteBase {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3832#discussion_r22447232
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -255,6 +255,45 @@ class CheckpointSuite extends TestSuiteBase {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3832#discussion_r22447211
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -255,6 +255,45 @@ class CheckpointSuite extends TestSuiteBase {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3832#discussion_r22447201
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -255,6 +255,45 @@ class CheckpointSuite extends TestSuiteBase {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3832#discussion_r22447198
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -255,6 +255,45 @@ class CheckpointSuite extends TestSuiteBase {
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3832#issuecomment-68662296
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3832#issuecomment-68662294
[Test build #25040 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25040/consoleFull)
for PR 3832 at commit
[`36eaf35`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3832#issuecomment-68659288
[Test build #25040 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25040/consoleFull)
for PR 3832 at commit
[`36eaf35`](https://githu
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3832#issuecomment-68659162
Alright, I've pushed one more commit that explains the confusing
`transform()` use and refers to
https://issues.apache.org/jira/browse/SPARK-5079. Feel free to fix up
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3832#issuecomment-68616374
The new test added here always failed before the fix in this patch. I
agree that it's a bit convoluted; it would be better to catch this bug in the
original `saveAs*`
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3832#issuecomment-68616268
Yes, this is hardly a valid used case of `transform`. I took a look at the
unit test, and I think I know why it works. But this unit test is very very
obfuscated, and hard t
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3832#issuecomment-68582508
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3832#issuecomment-68582506
[Test build #25003 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25003/consoleFull)
for PR 3832 at commit
[`6485cf8`](https://gith
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3832#issuecomment-68580794
@tdas I've updated this PR and added a test case. My test case uses calls
inside of a `transform()` call to emulate what Streaming's `saveAsHadoopFiles`
operation does
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3832#issuecomment-68580537
[Test build #25003 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25003/consoleFull)
for PR 3832 at commit
[`6485cf8`](https://githu
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/3832
[SPARK-4835] Disable validateOutputSpecs for Spark Streaming jobs
This patch disables output spec. validation for jobs launched through Spark
Streaming, since this interferes with checkpoint recov
18 matches
Mail list logo
