criccomini opened a new issue, #831:
URL: https://github.com/apache/arrow-rs-object-store/issues/831
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
SlateDB currently has a `RetryingObjectStore` that handles `ObjectStore`
retries. The `object_store` retry mechanisms are unsuitable for us. This is
something we've discussed in various GH issues in the past. We do something
like this:
```rs
#[inline]
fn should_retry(err: &object_store::Error) -> bool {
let retry = !matches!(
err,
object_store::Error::AlreadyExists { .. }
| object_store::Error::Precondition { .. }
| object_store::Error::NotModified { .. }
| object_store::Error::NotFound { .. }
| object_store::Error::NotImplemented { .. }
| object_store::Error::NotSupported { .. }
);
if !retry {
debug!("not retrying object store operation [error={:?}]", err);
}
retry
}
```
This approach is somewhat similar to what Lance does
(https://github.com/apache/arrow-rs-object-store/issues/737).
The problem we have right now is there's no good way for us to handle
retries for multipart PUTs. We currently inherit `object_store`'s behavior for
those writes. This behavior is insufficient
(https://github.com/slatedb/slatedb/issues/1956).
**Describe the solution you'd like**
We've been wrestling a bit with the best way to handle this. The current
`object_store` API makes it impossible for us to retry a multipart upload
without buffering the entire byte object and retrying from scratch (as far as I
can tell--please correct me if I'm wrong). I am seeking advice/suggestions at
this point.
There are a few challenges here:
- The multipart puts are async. We don't see a part failure until later.
- We don't know which parts failed AFAICT.
- There's no high-level API for us to retry a failed part. I see
`MultipartStore::put_part(.., part_idx, data)`, but that isn't exposed at the
`ObjectStore` level.
What do you think? Should `object_store` to support custom retry policies
for multipart uploads (or for uploads in general)? Should users be allowed to
retry a specific part write?
It's quite possible I'm misunderstanding something here, so LMK if that's
the case.
**Describe alternatives you've considered**
Living with what we have, which can halt SlateDB databases in certain
scenarios.
**Additional context**
Seel GH links above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
