jkosh44 commented on PR #7301:
URL: https://github.com/apache/arrow-rs/pull/7301#issuecomment-2734584034
So I created a brand new bucket and ran the following test locally about
1000 times on `main`, and it passed every time.
```rust
use std::fs::File;
use std::io::Read;
use object_store::gcp::GoogleCloudStorageBuilder;
use object_store::multipart::MultipartStore;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let store =
GoogleCloudStorageBuilder::new().with_bucket_name("multipart-complete-header-test").with_application_credentials("path/to/my/credentials.json").build().unwrap();
let file_path = "path/to/some/local/file";
let file_contents = {
let mut buf = Vec::new();
File::open(file_path)
.unwrap()
.read_to_end(&mut buf)
.unwrap();
buf
};
let blob_path = "multi-test/hello.txt".into();
let id = store.create_multipart(&blob_path).await.unwrap();
let mut part_ids = Vec::new();
let part_id = store
.put_part(&blob_path, &id, 0, file_contents.into())
.await
.unwrap();
part_ids.push(part_id);
store
.complete_multipart(&blob_path, &id, part_ids)
.await
.unwrap();
// Sleep to avoid rate limiting.
tokio::time::sleep(tokio::time::Duration::from_millis(1000)).await;
Ok(())
}
```
It does seem very odd that the docs clearly state that content-length is
required, but we are not setting the content-length. My best guess as to why I
was receiving a 411 is that GCP had a partial roll-out of new API servers that
were validating content-length ... but that seems a bit far-fetched.
I did try the same test with this branch, and it also works fine. So I'll
leave it up to you if you think it's worth merging. In terms of unit tests, is
there somewhere where we are already unit testing multipart uploads? It seems
difficult to test without refactoring s.t. we generate and execute the request
in separate methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
