Kino1994 opened a new pull request, #58212:
URL: https://github.com/apache/spark/pull/58212
### What changes were proposed in this pull request?
Use a managed temp directory for the `checkpointLocation` of the `e2e
stateless` test in `KafkaRealTimeIntegrationSuite`:
```scala
- .option("checkpointLocation",
Files.createTempDirectory("some-prefix").toFile.getName)
+ .option("checkpointLocation", Utils.createTempDir().getCanonicalPath)
```
`java.nio.file.Files` was only used by that line, so its import is dropped.
### Why are the changes needed?
`Files.createTempDirectory(...)` returns the full path of the directory it
creates, but `.toFile.getName` keeps only the last segment of it:
```
full path : /tmp/some-prefix7700179876677442263
getName() : some-prefix7700179876677442263
isAbsolute : false
```
So the option is a relative path, which Spark resolves against the working
directory. Each run of the suite leaks two directories, neither of which is
cleaned up:
- the temp directory that was created and then never used, under
`java.io.tmpdir`
- the checkpoint itself, written into the module directory as
`connector/kafka-0-10-sql/some-prefix<N>`
The second one is not covered by `.gitignore`, so it shows up as untracked
in `git status` for anyone who runs the module's tests. After two full runs of
`sql-kafka-0-10` locally:
```
./connector/kafka-0-10-sql/some-prefix6311020086553351455
./connector/kafka-0-10-sql/some-prefix9434080927785378187
./target/tmp/some-prefix6311020086553351455
./target/tmp/some-prefix9434080927785378187
```
The suffixes pair up, one pair per run.
`Utils.createTempDir()` is what the rest of the module already uses for
this: it is absolute, lives under `java.io.tmpdir`, and is deleted on JVM exit.
The other five `checkpointLocation` usages in `connector/kafka-0-10-sql` all
pass an absolute path from a managed temp directory; this line was the only one
that did not.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
Ran `KafkaRealTimeIntegrationSuite` and confirmed the test still passes and
that no `some-prefix*` directory is left behind afterwards, in the module
directory or under `target/tmp`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
