ahmedabu98 opened a new pull request, #41248: URL: https://github.com/apache/beam/pull/41248
With the side input table cache on, writers rebuild each table's `FileIO` from a `SerializableTableSpec`. That had three problems: 1. **Vended credentials were dropped.** The spec stored the `FileIO` as `FileIOParser` JSON, which keeps only `io-impl` and the properties. Credentials a REST catalog vends through `SupportsStorageCredentials` were lost, so writes fell back to the workers' default credentials. 2. **Fresh credentials were ignored.** The cache replaced a table's spec only when the table's metadata changed. A poll of an unchanged table brought new credentials, and the cache threw them away. 3. **`FileIO`s leaked.** Each side input update decodes new copies of every spec, and each copy built its own `FileIO`. Nothing closed them. With expiring credentials, `GCSFileIO` and `S3FileIO` also schedule a refresh that keeps every old `FileIO` alive. ### Changes - `SerializableTableSpec` has a new nullable field, `storageCredentialJsons` (field 13). It holds each credential as Iceberg `CredentialParser` JSON, and `getFileIO` sets them on the rebuilt `FileIO`. - `AccumulateTableMetadataMapDoFn` now keeps the cached spec only if it's newer than the poll's. Unchanged tables pick up the latest credentials. - Specs with the same `FileIO` JSON and credentials share one `FileIO`. A `Cleaner` closes it once no spec that uses it is reachable, which also cancels its credential refresh. Writers keep their spec reachable while they write, so a `FileIO` can't close mid-write. ### Compatibility The new field is appended and nullable, so specs encoded before this change still decode, with the field null. Rolling back fails if the cache already holds specs with credentials. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
