jianvora opened a new issue, #877:
URL: https://github.com/apache/arrow-rs-object-store/issues/877
**Describe the bug**
With `object_store` 0.14.2, `AmazonS3`'s list request is signed without the
`ClientOptions` default headers. Other requests (`request`, `get_request`,
`bulk_delete_request`, `complete_multipart`) copy
`client_options.get_default_headers()` into the request before SigV4 signing
(the fix for #484). `list_request` (`src/aws/client.rs`) builds and signs its
request without them.
reqwest then merges the client's default headers into the request at send
time (`ClientBuilder::default_headers`). So a LIST carries every default
header, but none of them appears in `SignedHeaders`. For any `x-amz-*` default
header, S3 answers `403 AccessDenied` with `HeadersNotSigned`, the same failure
#708 reports for `CompleteMultipartUpload`. An `x-amz-*` default header such as
`x-amz-expected-bucket-owner` therefore works for GET and PUT but breaks every
list.
**To Reproduce**
This runs against a local TCP listener that prints what each request
carried. No AWS account is needed.
```rust
use std::io::{BufRead, BufReader, Write};
use std::net::TcpListener;
use std::sync::mpsc;
use futures::TryStreamExt;
use object_store::aws::AmazonS3Builder;
use object_store::path::Path;
use object_store::{ClientOptions, ObjectStore, ObjectStoreExt};
fn main() {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
let (tx, rx) = mpsc::channel::<(String, Vec<String>)>();
std::thread::spawn(move || {
for stream in listener.incoming() {
let mut stream = stream.unwrap();
let mut reader = BufReader::new(stream.try_clone().unwrap());
let mut request_line = String::new();
reader.read_line(&mut request_line).unwrap();
let mut headers = Vec::new();
loop {
let mut line = String::new();
reader.read_line(&mut line).unwrap();
if line.trim().is_empty() {
break;
}
headers.push(line.trim().to_owned());
}
let body = if request_line.contains("list-type=2") {
"<ListBucketResult><IsTruncated>false</IsTruncated></ListBucketResult>"
} else {
"hello"
};
write!(
stream,
"HTTP/1.1 200 OK\r\nContent-Length: {}\r\nLast-Modified:
Thu, 01 Jan 2026 00:00:00 GMT\r\nETag: \"e\"\r\nConnection:
close\r\n\r\n{body}",
body.len()
)
.unwrap();
tx.send((request_line.trim().to_owned(), headers)).unwrap();
}
});
let mut defaults = object_store::HeaderMap::new();
defaults.insert("x-amz-expected-bucket-owner",
"111122223333".parse().unwrap());
let store = AmazonS3Builder::new()
.with_bucket_name("bucket")
.with_region("us-east-1")
.with_endpoint(format!("http://{addr}";))
.with_access_key_id("AKIDEXAMPLE")
.with_secret_access_key("secret")
.with_client_options(
ClientOptions::new()
.with_allow_http(true)
.with_default_headers(defaults),
)
.build()
.unwrap();
let rt = tokio::runtime::Runtime::new().unwrap();
rt.block_on(async {
store.get(&Path::from("key")).await.unwrap().bytes().await.unwrap();
let _: Vec<_> =
store.list(Some(&Path::from("p"))).try_collect().await.unwrap();
});
for _ in 0..2 {
let (line, headers) = rx.recv().unwrap();
let sent = headers
.iter()
.any(|h|
h.to_ascii_lowercase().starts_with("x-amz-expected-bucket-owner:"));
let signed = headers
.iter()
.find(|h| h.to_ascii_lowercase().starts_with("authorization:"))
.is_some_and(|a| a.contains("x-amz-expected-bucket-owner"));
println!("{line}\n header sent: {sent}, in SignedHeaders:
{signed}");
}
}
```
Dependencies: `object_store = { version = "=0.14.2", features = ["aws"] }`,
`tokio = { version = "1", features = ["rt-multi-thread", "macros"] }`, `futures
= "0.3"`.
Output:
```
GET /bucket/key HTTP/1.1
header sent: true, in SignedHeaders: true
GET /bucket?list-type=2&prefix=p%2F HTTP/1.1
header sent: true, in SignedHeaders: false
```
**Expected behavior**
The list request copies the default headers into the request before signing,
as the other request paths do, so they are signed on LIST too.
**Additional context**
- The `test_default_headers_signed_*` tests in `src/aws/client.rs` cover
request, bulk delete, GET and complete multipart, but not list.
- Any request path that builds its request without the defaults (presigned
URLs aside) has the same gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
