adrien-grl opened a new pull request, #51631:
URL: https://github.com/apache/arrow/pull/51631
### Rationale for this change
Closing an output stream always commits what was written, so a job that
fails halfway through still leaves a partial object on S3. Arrow C++ already
has `Abort()` for this, and the S3 stream implements it by aborting the
multipart upload, but it wasn't reachable from Python.
### What changes are included in this PR?
This adds `NativeFile.abort()`, which calls `FileInterface::Abort()` the
same way `close()` calls `Close()`. The example from the issue now works:
```python
with fs.open_output_stream("bucket/key") as f:
try:
do_something_that_may_explode(f)
except Exception:
f.abort()
raise
```
### Are these changes tested?
Yes. `test_open_output_stream_abort` runs on all the filesystem fixtures,
with and without compression and buffering. It checks that nothing is written
on S3, and that the mock filesystem sees an abort rather than a close.
`test_s3_output_stream_abort_after_part_upload` aborts after a 10 MiB part has
already been uploaded, which I don't think the C++ tests cover. There are also
two small tests in `test_io.py` for aborting in-memory streams.
Backends other than S3 don't discard anything on abort today so the tests
are pretty light for the other backends. I ran it against minio, azurite and
the GCS testbench. Local, GCS and fsspec keep the written data, since their
`Abort()` just closes. Azure leaves an empty blob, because the blob is created
when the stream is opened. Those seem worth separate issues.
`test_io.py` and `test_fs.py` pass locally with S3, Azure and GCS enabled.
### Are there any user-facing changes?
Yes, `NativeFile.abort()` is new.
### Was AI used for this PR?
In accordance to the [AI generation
guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code),
please disclose below whether and how AI was used in this PR.
I used Claude Code to write the code, the tests and this description, and to
run the tests locally.
**PR code and description written by:**
- [X] Human
- [x] AI
**Reviewed before submission by:**
- [X] Human
- [x] AI
- [ ] Not reviewed
* GitHub Issue: #40502
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
