senzzzi opened a new pull request, #817:
URL: https://github.com/apache/arrow-rs-object-store/pull/817
# Which issue does this PR close?
Closes #281.
# Rationale for this change
`PutPayload` currently only supports collections of `Bytes`. As a result,
uploading a file through `put` or `put_opts` requires loading the entire file
into memory or using a multipart upload.
This change allows callers to stream a payload with a known content length
through a single PUT request. File-backed payloads are a primary use case.
# What changes are included in this PR?
- Extends `PutPayload` to support replayable streaming bodies with a known
content length.
- Adds `PutPayload::from_stream` for constructing streaming payloads.
- Adds `PutPayload::from_file`, which reads files in 16 KiB chunks by
default.
- Adds `PutPayload::from_file_with_chunk_size` for configuring the file
chunk size.
- Updates HTTP request bodies to poll payload streams asynchronously.
- Recreates the stream for every request attempt so retryable uploads can
resend the complete payload.
- Updates S3 signing and checksum calculation to consume streaming payloads.
- Updates the local filesystem and in-memory backends to accept streaming
payloads.
- Adds regression tests covering:
- the default 16 KiB file chunk size;
- replaying cloned streaming request bodies; and
- retrying an upload using a file-backed streaming payload.
Validation performed:
- `cargo test`
- `cargo test --all-features`
- `cargo clippy --all-targets --all-features -- -D warnings`
- compilation with only the `fs` feature
- compilation without default features
# Are there any user-facing changes?
Yes. Callers can now upload a file without first loading the complete file
into memory:
```rust
let payload = PutPayload::from_file(file_path).await?;
store.put(&location, payload).await?;
```
Custom streaming payloads can be created with PutPayload::from_stream. The
stream factory must produce a new stream from the beginning for every
invocation, and the
supplied content length must exactly match the number of bytes yielded.
Existing byte-backed PutPayload construction and iteration behavior is
unchanged. Synchronous APIs such as iter, AsRef<[Bytes]>, and conversion into
Bytes are not supported for streaming payloads; callers should use
PutPayload::stream or PutPayload::bytes instead.
## API compatibility consideration
`PutPayload` remains a public struct backed by a private enum so that
existing byte-backed construction and usage remain source compatible:
```rust
enum PutPayloadInner {
Bytes(Arc<[Bytes]>),
Streaming {
factory: StreamFactory,
content_length: usize,
},
}
```
There is an API-design limitation around the existing synchronous accessors.
Methods and trait implementations such as PutPayload::iter, AsRef<[Bytes]>,
IntoIterator, and conversion into Bytes cannot synchronously represent an
asynchronous, fallible stream.
In this implementation, their existing behavior is unchanged for byte-backed
payloads, but calling them with a streaming payload panics. Streaming-aware
implementations should instead use PutPayload::stream or PutPayload::bytes.
Changing the existing synchronous APIs to return Option or Result would
avoid the panic but would be a breaking public API change. Returning an empty
iterator for streaming payloads was rejected because it could silently result
in incomplete uploads.
Feedback from maintainers on the preferred long-term API shape would be
appreciated. Possible alternatives include:
- accepting the documented limitation for synchronous APIs;
- changing these APIs in a future breaking release; or
- introducing streaming uploads through a separate payload type or method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
