sundapeng commented on code in PR #10350:
URL: https://github.com/apache/paimon/pull/10350#discussion_r4377903020
##########
paimon-common/src/main/java/org/apache/paimon/rest/RESTTokenFileIO.java:
##########
@@ -247,13 +297,13 @@ private FileIOWithToken fileIOWithToken(long
minimumValidityMillis) throws IOExc
FileIO fileIO = FILE_IO_CACHE.getIfPresent(currentToken);
if (fileIO != null) {
- return new FileIOWithToken(fileIO, currentToken);
+ return new FileIOWithToken(viewOf(fileIO), currentToken);
Review Comment:
Thanks, confirmed and fixed in 84363ffb61. The static FileIO cache was keyed
by the token alone, while the FileIO is built from the catalog options merged
with the token, and routing can come from the catalog options alone.
The key is now the token together with an immutable copy of the catalog
options, taken once per `RESTTokenFileIO`, and the FileIO is built from that
same copy. The catalog's explicit `dlf.oss-endpoint` and `io-cache.enabled` are
already merged into the token at refresh, so they are covered too. Copies of
one catalog's `RESTTokenFileIO`, including deserialized ones on workers, still
share one FileIO.
I treated it as one class: anything that builds or caches an endpoint-bound
client must use the effective options (catalog options merged with the token,
then the catalog's overrides) and be keyed by everything that changes the
backend. What I checked on the Java side:
- `RESTTokenFileIO.FILE_IO_CACHE`: fixed as above.
- `JindoFileIO`'s static file-system cache is keyed by the full Hadoop
options, endpoint included, and REST turns it off; per-target file systems
belong to one `JindoFileIO` and are shared only by its views.
`HadoopCompliantFileIO`, `ResolvingFileIO` and `UriReaderFactory` caches are
per instance. `CachingFileIO`'s shared manager caches immutable whitelisted
files by path, not by endpoint.
- The token is kept per `RESTTokenFileIO` instance; nothing shares an
already-overridden token across catalogs.
- Lance and Vortex take `hadoopOptions(path, op)`, which is the origin's
options of the effective configuration, so they follow the fix as well.
- `UriReaderFactory` builds its FileIO from the catalog context and reads
through its origin-only view; with `JindoFileIO` that is the effective origin.
A FileIO that does not support routing (paimon-oss) uses `fs.oss.endpoint` for
the table and for descriptors alike, as before this PR.
Tests:
`RESTTokenFileIOTest#testFileIOsOfOneTokenAreCachedPerCatalogOptions`, and
`JindoIoCacheRoutingTest#testTokenFileIOsOfOneTokenKeepTheirCatalogEndpoints`,
which builds two `RESTTokenFileIO`s with one token and different catalog-only
origin and target through the real REST cache and `JindoFileIO` configuration,
and checks reads, writes and the Lance options of each. Both fail without the
change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
