guangyu-yang-rokt opened a new pull request, #12827: URL: https://github.com/apache/gluten/pull/12827
## What changes were proposed in this pull request? Fixes #12701: make native scans work for Iceberg tables read through a REST catalog that vends credentials (`X-Iceberg-Access-Delegation: vended-credentials`, e.g. Apache Polaris). `loadTable` returns credentials scoped to one table and only the JVM `FileIO` ever sees them. The native scan path receives file paths only, so Velox's S3 client falls back to the process credential chain — which by design has no access to the warehouse — and every native `TableScan` of such a table dies with S3 403 (`S3ReadFile.cpp preadInternal: Failed to get S3 object due to: 'Access denied'`) while the same query reads fine on vanilla Spark. **JVM side.** At split planning, read the scan table's `FileIO.properties()`; when a vended access-key/secret pair is present, carry it (plus session token/expiry/endpoint/region companions) together with the table location in a new table-scoped `ReadRel.LocalFiles.read_properties` map. A `LocalFiles` is single-table by construction, so table granularity is split granularity. `FileIO.properties()` is implemented by `S3FileIO`/`ResolvingFileIO` and defaults to throwing, which is treated as "no credentials". Tables whose files the process credentials can read (e.g. Glue-catalog tables) attach nothing and are untouched. **Native side.** `parseScanSplitInfo` copies the map onto `SplitInfo`, and `WholeStageResultIterator` unions the task's scans into a `GlutenS3TokenProvider` installed on the `QueryCtx` as the file system token provider. Resolution is a segment-boundary-safe longest-prefix match of the file path against the normalized table locations, so a query joining two vended-credential tables that live in the same bucket cannot mix them up, and a table nested inside another table's location still gets its own credentials. `equals()`/`hash()` cover all credentials because they key Velox's file handle cache — a re-vended credential set can never be served a handle opened with the set it replaced. When no scan carries credentials the provider is `nullptr`, i.e. behaviour is unchanged. Velox already threads `ConnectorQueryCtx::fsTokenProvider()` into every data-file and delete-file open through `FileHandleKey` and `FileOptions`. **New config.** `spark.gluten.sql.columnar.iceberg.enableVendedCredentials` (default `true`). When disabled, scans of tables read with vended credentials fail native validation and fall back to vanilla Spark, which reads them correctly through the JVM `FileIO` — a working escape hatch rather than a degraded one. ### Dependency on a Velox PR The native half needs the S3 file system to consume `FileOptions::tokenProvider`, which no file system does today (only ABFS and GCS have token-provider paths, and neither uses `FileOptions`). That is [facebookincubator/velox#18570](https://github.com/facebookincubator/velox/pull/18570), which adds `S3AccessTokenKey`/`S3AccessToken` and a per-credential client cache in `S3FileSystem`. The second commit here (`[MINOR]`) points `UPSTREAM_VELOX_PR_ID` at that PR so CI can build this one, and **must be reverted before merge**, once the Velox change is merged and the pin is advanced past it. Happy to split this PR into the JVM half and the native half if you would rather land them separately. ### Notes for reviewers - Credentials ride the substrait split payload to executors, i.e. the same trust plane as the broadcast Hadoop conf. Say the word if you would prefer the config to default to `false` (opt-in) instead. - Credentials are snapshotted on the driver, so a scan has to start within the lifetime of the vended credentials; executors cannot re-vend. A 403 on expiry stays a retriable task failure. Refresh-on-open is a follow-up — this is the same staging `apache/datafusion-comet` used for the same problem (comet#3523 static extraction, then comet#4309 pluggable refresh). ## How was this patch tested? New unit tests, all passing locally (`-Pbackends-velox,spark-3.5,iceberg`, JDK 17, scalastyle and spotless gates live): - `GlutenIcebergSourceUtilSuite` (3 tests) — the credential set is extracted with its location and companions and nothing else; a table without a vended access-key/secret pair extracts nothing; the session token is optional. - `IcebergLocalFilesNodeReadPropertiesTest` (2 tests) — the map round-trips into `ReadRel.LocalFiles.read_properties`, and a split without credentials emits no `read_properties`. - `GlutenS3TokenProviderTest` (5 tests, `ENABLE_S3`) — no provider without credentials or from an incomplete set; per-table resolution for two tables in one bucket; longest-prefix wins for nested locations; a string-prefix table name is not a match; identity covers all credentials; scheme normalization. Also covered by the Velox PR's Minio-backed tests, which prove that provider-supplied credentials are the ones used to open the file. End to end, this patch set (backported to 1.6.0) has been running Iceberg reads against Apache Polaris with STS credential vending on Spark 3.5.2 in production, on both arm64 and x86_64: native `TableScan` of vended-credential tables reads with the vended credentials instead of 403ing, and row-level parity against vanilla Spark holds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
