YuangGao opened a new pull request, #3179: URL: https://github.com/apache/iceberg-rust/pull/3179
## Which issue does this PR close? - Part of #2977. ## What changes are included in this PR? OpenDAL turns one caller buffer into one request, and `ParquetWriter` hands over a whole row group per `write` call. A 128 MiB row group therefore became a single `UploadPart` racing the hard-coded 10s IO timeout, and since every retry re-sends the same oversized request, the write fails deterministically rather than flakily. Bound the request size so it follows configuration instead of the caller's buffer: - Add `s3.multipart.part-size-bytes`, default 32 MiB (matching Java `S3FileIOProperties.MULTIPART_SIZE_DEFAULT`), rejected below the 5 MiB S3 minimum for a non-final part. - `OpenDalStorage::S3` applies it through `write_options` / `writer_options`. Other backends keep OpenDAL's defaults. ## Are these changes tested? - Unit tests for property parsing: default, override, 5 MiB boundary, non-numeric input. - Integration test against MinIO asserting one `FileWrite::write` is split into the configured number of upload parts, counted from the multipart ETag suffix. - Verified with `mc admin trace`: a 160 MiB Parquet write previously issued one request, now issues 5 x 32 MiB parts plus a remainder. ## AI Disclosure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
