ntjohnson1 opened a new pull request, #790:
URL: https://github.com/apache/arrow-rs-object-store/pull/790
# Which issue does this PR close?
Closes #706
I included a commit with the exemplar I was using to verify
arrow-rs-object-store works with HF buckets (to demonstrate things work if you
want to try it yourself)
# Rationale for this change
With hugging face buckets S3 compatible API you can create a builder via:
```rust
let mut builder = AmazonS3Builder::new()
.with_endpoint(endpoint)
.with_region(region)
.with_bucket_name(bucket)
.with_access_key_id(env_req("AWS_ACCESS_KEY_ID"))
.with_secret_access_key(env_req("AWS_SECRET_ACCESS_KEY"))
.with_disable_bulk_delete(disable_bulk_delete);
```
However, when I did this and tried connecting to a bucket I was getting
issues with the redirect.
object_store's SigV4 signer inserted an explicit Host header on every
request. reqwest carries user-set headers across redirects, so when an
S3-compatible gateway answers GetObject with a cross-host 302 ( e.g. the
Hugging Face gateway: s3.hf.co redirecting to its Xet CDN:
cas-bridge.xethub.hf.co) the stale Host: s3.hf.co header poisoned the
redirected request and the download failed with a connection error. list/head
worked (no redirect); every get failed.
# What changes are included in this PR?
Host is still part of the SigV4 signature, but it's now added to a throwaway
copy of the headers used only to compute the signature, instead of being pinned
on the outgoing request. The actual Host header is left to hyper, which derives
it from the URL and regenerates it on redirect. The produced signature is
identical, so AWS S3 and all other providers are unaffected.
# Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
