GitHub user uncleGen opened a pull request:
https://github.com/apache/spark/pull/17167
[SPARK-19822][TEST] CheckpointSuite.testCheckpointedOperation: should not
check checkpointFilesOfLatestTime by the PATH string.
## What changes were proposed in this pull request?
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73800/testReport/
```
org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed
to eventually never
returned normally. Attempted 617 times over 10.003740484 seconds. Last
failure message: 8 did
not equal 2.
```
the check condition is:
```
val checkpointFilesOfLatestTime =
Checkpoint.getCheckpointFiles(checkpointDir).filter {
_.toString.contains(clock.getTimeMillis.toString)
}
// Checkpoint files are written twice for every batch interval. So assert
that both
// are written to make sure that both of them have been written.
assert(checkpointFilesOfLatestTime.size === 2)
```
the path string may contain the `clock.getTimeMillis.toString`, like:
```
file:/root/dev/spark/assembly/CheckpointSuite/spark-20035007-9891-4fb6-91c1-cc15b7ccaf15/checkpoint-500
file:/root/dev/spark/assembly/CheckpointSuite/spark-20035007-9891-4fb6-91c1-cc15b7ccaf15/checkpoint-1000
file:/root/dev/spark/assembly/CheckpointSuite/spark-20035007-9891-4fb6-91c1-cc15b7ccaf15/checkpoint-1500
file:/root/dev/spark/assembly/CheckpointSuite/spark-20035007-9891-4fb6-91c1-cc15b7ccaf15/checkpoint-2000
file:/root/dev/spark/assembly/CheckpointSuite/spark-20035007-9891-4fb6-91c1-cc15b7ccaf15/checkpoint-2500
file:/root/dev/spark/assembly/CheckpointSuite/spark-20035007-9891-4fb6-91c1-cc15b7ccaf15/checkpoint-3000
file:/root/dev/spark/assembly/CheckpointSuite/spark-20035007-9891-4fb6-91c1-cc15b7ccaf15/checkpoint-3500.bk
file:/root/dev/spark/assembly/CheckpointSuite/spark-20035007-9891-4fb6-91c1-cc15b7ccaf15/checkpoint-3500
-------------------------------------------------------^^^^-------------------------------------------
```
so we should only check the filename, but not the while path.
## How was this patch tested?
Jenkins.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/uncleGen/spark flaky-CheckpointSuite
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17167.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17167
----
commit 72f1963a36f9f1abfe8ca10d30b01f52c2281d82
Author: uncleGen <husty...@gmail.com>
Date: 2017-03-03T10:11:52Z
flaky CheckpointSuite test failure
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
