Github user JoshRosen closed the pull request at:
https://github.com/apache/spark/pull/3687
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is e
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-69080052
I'm going to close this for now; I'll open separate, smaller PRs for the
remaining test cleanup.
---
If your project is set up for it, you can reply to this email and
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r22269847
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,34 +278,45 @@ class CheckpointSuite extends TestSuiteBase {
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-68094079
Just realized that my last comment was a bit confusing, since SPARK-1600 is
not related to the FileInputStream ManualClock fix. I'll file a new
improvement JIRA to cov
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-68091244
I agree that it's a good idea to split this up.
For starters, I'm going to try splitting off only the fix for the
FileInputDStream test (SPARK-1600), since that
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-68082434
At a high-level i think, let spilt this PR into two PRs
1. Fixing saveAsHadoopFiles
2. Removing thread loops
This would allow the discussion of one to procee
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r22265904
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala ---
@@ -104,6 +106,77 @@ class TestOutputStreamWithPartitions[T:
ClassTag
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r22265867
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala ---
@@ -104,6 +106,77 @@ class TestOutputStreamWithPartitions[T:
ClassTag](par
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r22265799
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -281,34 +278,45 @@ class CheckpointSuite extends TestSuiteBase {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r22265787
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -46,8 +46,6 @@ class CheckpointSuite extends TestSuiteBase {
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r22265770
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala
---
@@ -639,7 +639,7 @@ class BasicOperationsSuite extends TestSuiteBas
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r22265754
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/util/Clock.scala ---
@@ -59,9 +59,11 @@ class SystemClock() extends Clock {
private[streaming]
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r22265733
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
---
@@ -74,12 +74,15 @@ class FileInputDStream[K: ClassTag, V: Clas
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67273407
[Test build #24524 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24524/consoleFull)
for PR 3687 at commit
[`1304776`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67273413
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67272979
[Test build #24523 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24523/consoleFull)
for PR 3687 at commit
[`520bade`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67272984
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r21949212
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -431,15 +454,15 @@ class CheckpointSuite extends TestSuiteBase {
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r21949136
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala ---
@@ -286,22 +359,23 @@ trait TestSuiteBase extends FunSuite with
Befor
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r21949039
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -46,8 +46,6 @@ class CheckpointSuite extends TestSuiteBase {
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67268177
[Test build #24524 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24524/consoleFull)
for PR 3687 at commit
[`1304776`](https://githu
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67268086
Alright, I think I fixed up a few problems in my first version of the
SPARK-1600 fix, so hopefully it works now.
For those who are interested in the details:
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67267461
[Test build #24523 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24523/consoleFull)
for PR 3687 at commit
[`520bade`](https://githu
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67265698
It turns out that my current use of `file.setLastModified` is brittle since
the underlying filesystem may only support one-second resolution:
```
* Al
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67239409
This latest run failed three tests:
- org.apache.spark.streaming.CheckpointSuite.recovery with file input
stream
- org.apache.spark.streaming.FailureSuite
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67137400
[Test build #24495 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24495/consoleFull)
for PR 3687 at commit
[`ee8c8f8`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67137409
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r21884849
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala ---
@@ -238,14 +190,17 @@ class InputStreamsSuite extends TestSuiteBas
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67130485
[Test build #24495 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24495/consoleFull)
for PR 3687 at commit
[`ee8c8f8`](https://githu
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r21884670
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala ---
@@ -305,7 +305,7 @@ class CheckpointSuite extends TestSuiteBase {
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r21884615
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
---
@@ -74,12 +74,14 @@ class FileInputDStream[K: ClassTag, V:
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-67130169
I pushed some more changes (this is still WIP). In addition to our use of
Thread.sleep, I think I should also look out for cases where streaming directly
calls System.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-66892579
This latest round of test failures was due to the flaky
`WriteAheadLogBackedBlockRDDSuite` tests
([SPARK-4826](https://issues.apache.org/jira/browse/SPARK-4826)). I'm
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r21787343
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala ---
@@ -104,6 +106,49 @@ class TestOutputStreamWithPartitions[T:
ClassTag](par
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r21787336
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala ---
@@ -104,6 +106,49 @@ class TestOutputStreamWithPartitions[T:
ClassTag](par
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3687#discussion_r21787328
--- Diff: docs/configuration.md ---
@@ -709,7 +709,9 @@ Apart from these, the following properties are also
available, and may be useful
If set to true,
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-66871584
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-66871580
[Test build #24435 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24435/consoleFull)
for PR 3687 at commit
[`12635b4`](https://gith
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-66871384
@JoshRosen This is a wonderful and much-required refactoring. I havent been
able to see it in detail yet, will do when I reach home.
---
If your project is set up for it,
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-66870761
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-66870757
[Test build #24434 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24434/consoleFull)
for PR 3687 at commit
[`3db335f`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-66870136
[Test build #24435 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24435/consoleFull)
for PR 3687 at commit
[`12635b4`](https://githu
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-66869474
[Test build #24434 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24434/consoleFull)
for PR 3687 at commit
[`3db335f`](https://githu
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3687#issuecomment-66865631
/cc @tdas. This is a work-in-progress towards removing most of the
Thread.sleep() calls. I'm making slow-but-steady progress; would love your
feedback + any suggestio
44 matches
Mail list logo
