williamhyun opened a new pull request, #17457: URL: https://github.com/apache/iceberg/pull/17457
## Summary Adds support for reading object-store files over catalog-vended HTTP(S) locations (e.g. pre-signed URLs) without requiring native cloud credentials on the reader, and wires it into `S3FileIO`. - **Core:** Introduces `HttpUrlSupport`, `HTTPInputFile`, and `HTTPInputStream` — a seekable, range-readable HTTP(S) input path built on the `httpclient5` dependency `iceberg-core` already depends on (also used by `org.apache.iceberg.rest.HTTPClient`), so no new third-party dependency is added. Reads are served via chunked range GETs (4 MB chunks), with retries on transient network/server errors during those range fetches; content length is probed via `Range: bytes=0-0` when unknown, since pre-signed GET URLs typically reject `HEAD`. - **AWS:** `S3FileIO#newInputFile(String)` / `newInputFile(String, long)` now short-circuit to `HttpUrlSupport` when the location is an `http://` or `https://` URL, leaving `s3://`/`s3a://`/`s3n://` reads on the native, credentialed S3 client path unchanged. `S3FileIO#close()` now also closes the shared HTTP client. - **Tests:** Unit coverage for the dispatch decision (`TestS3FileIOHttpUrlDispatch`) and for `HTTPInputFile`/`HTTPInputStream` behavior against a real local `HttpServer` (`TestHTTPInputFile`, including multi-chunk reads that cross the 4 MB boundary); a MinIO-backed integration test (`TestS3FileIOPresignedUrls`) that mints real pre-signed URLs and proves the native S3 client is never invoked; and an optional live-AWS integration test (`TestS3FileIOLivePresignedUrls`) gated on AWS credential env vars. This is the `FileIO` building block for catalogs that vend a pre-signed URL as a file's `file-path` in scan planning, so readers can fetch bytes over HTTPS with auth encoded in the URL itself. Delete-file handling, scan-planning integration, and other cloud providers are out of scope for this PR (see below). ## Scope / compatibility notes 1. The catalog is expected to place a usable, already-signed URL directly in the file's location (`file-path`); the client uses that location unchanged as the fetch URL. There is no separate signing lookup or location-to-URL mapping in this PR. 2. Only `S3FileIO` is updated. `ResolvingFileIO` and the GCS/ADLS equivalents are unchanged — a catalog that wants this behavior today needs to vend `io-impl: S3FileIO` per-table via the REST `LoadTableResponse.config()` map (the specified mechanism for a catalog to select a `FileIO` for a table), rather than relying on `ResolvingFileIO`'s scheme-based dispatch. Extending the other cloud `FileIO`s to the same short-circuit pattern is a natural follow-up. 3. Positional deletes are not supported. 4. `SplitScanTasks` are already unsupported on the remote scan-plan read path, as are multiple deletion vectors in the same deletion file. These are known limitations of the remote scan-plan endpoint and cannot be fixed without spec changes. ## Test plan - [ ] `./gradlew :iceberg-core:test --tests org.apache.iceberg.io.http.TestHTTPInputFile` - [ ] `./gradlew :iceberg-aws:test --tests org.apache.iceberg.aws.s3.TestS3FileIOHttpUrlDispatch` - [ ] `./gradlew :iceberg-aws:integrationTest --tests org.apache.iceberg.aws.s3.TestS3FileIOPresignedUrls` (requires Docker; spins up MinIO via Testcontainers) - [ ] Optionally, with `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, and `AWS_TEST_BUCKET` set: `./gradlew :iceberg-aws:integrationTest --tests org.apache.iceberg.aws.s3.TestS3FileIOLivePresignedUrls` - [ ] Confirm `s3://` reads through `S3FileIO` remain unchanged (no HTTP short-circuit) — covered by `nativeS3ReadForS3Location` and `s3LocationUsesNativeClient` --- **AI Disclosure** - Model: Claude Opus 4.8 - Platform/Tool: Cursor - Human Oversight: fully reviewed - Prompt Summary: Implement a mixin-based HTTP(S) read path for S3FileIO so readers can fetch bytes from catalog-vended pre-signed URLs without native S3 credentials, plus unit/integration/live-AWS test coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
