stantheman0128 opened a new pull request, #7638:
URL: https://github.com/apache/datafusion-comet/pull/7638
## Which issue does this PR close?
Closes #7594.
## Rationale for this change
A `spark.local.dir` directory named `local.data` made native shuffle writes
fail with `shuffle write error: Os { code: 2, kind: NotFound, message: "No such
file or directory" }`. The temp path was built with
`dataFile.getPath.replace(".data", ".data.tmp")`, so the directory itself was
rewritten to `local.data.tmp` and the native writer opened a path whose parent
did not exist.
A failed map task also left that temp file on disk. `stop(false)` did not
delete it.
## What changes are included in this PR?
The local temp path now comes from
`IndexShuffleBlockResolver.createTempFile`, which appends a unique suffix in
the same directory. `write()` deletes that file when the attempt fails, and
`stop(false)` deletes it if it is still tracked. A successful
`writeMetadataFileAndCommit` clears the tracked path first, so `stop(true)`
does not delete the committed data file.
## How are these changes tested?
`mvn test` from the repo root, skipping scalastyle and spotless because this
Windows checkout already fails those checks on files this PR does not change.
Suite filter: `CometNativeShuffleWriterSuite native shuffle`.
Before the fix:
```
- native shuffle temp file stays in a local dir named local.data *** FAILED
*** (58 milliseconds)
"local.data[.tmp]" did not equal "local.data[]"
(CometNativeShuffleWriterSuite.scala:91)
```
After the fix:
```
- native shuffle temp file stays in a local dir named local.data (44
milliseconds)
- native shuffle stop(false) deletes a tracked temp file (7 milliseconds)
Total number of tests run: 2
Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
```
The native library was not rebuilt. These two tests use the block resolver
and `stop`, not a native shuffle execution.
Written with AI assistance (Cursor, Grok).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]