sundapeng opened a new issue, #10526: URL: https://github.com/apache/paimon/issues/10526
### Search before asking - [x] I searched in the issues and found nothing similar. ### Motivation #10178 lets open OSS streams of `OSSFileIO` pick up refreshed REST data tokens: the OSS SDK asks a credentials provider for every request, and the provider reloads the token through `RESTTokenRefresher`. Readers and writers implemented in native code through JNI never ask Java for credentials after they open, so they still fail once the token they started with expires. On master at `2d4eb99f3cee8abf16f7fcf583ddc195828f08b1`: - **Lance** (`lance-core` 0.39.0): `LanceUtils.toLanceSpecified` unwraps `RESTTokenFileIO.fileIO()`, reads `fs.oss.accessKeyId`, `fs.oss.accessKeySecret` and `fs.oss.securityToken` from `HadoopOptionsProvider.hadoopOptions`, and passes them as static storage options to `LanceFileReader.open` / `LanceFileWriter.open`. - **Vortex** (`vortex-jni` 0.73.0): `VortexUtils` does the same and passes a static `session_token` to `DataSource.open` and `VortexWriter.create`. - **Jindo** (JindoSDK 6.9.1): with a REST data token, `JindoFileIO` configures `SimpleCredentialsProvider` with the static keys. As far as I can tell from `jindo-sdk`, the native core receives either static keys or an STS endpoint (`provider.endpoint` / `provider.format`) that it polls itself, and does not call back into a Java provider. After #10178, `RESTTokenFileIO.fileIO()` goes through `RESTTokenRefresher`, so the snapshot a native handle receives has at least min(1 hour, half of the lifetime the token had when it was loaded) left. A native read or write that stays open longer, such as a large Lance file write in a streaming job, fails with expired credentials. ### Solution Keep `RESTTokenRefresher` as the only refresh engine and let native code pull credentials from it, rather than calling the REST server from native code, which would need the REST authentication and signing there as well. Depending on what each library offers: 1. **Minimum validity at open (Paimon only).** `RESTTokenFileIO` can already resolve a delegate whose token outlives a requested duration, which #10178 uses for presigned URLs. Expose that to `LanceUtils` and `VortexUtils` so a native handle opens with a token that covers its expected lifetime. This narrows the window without closing it; readers could reopen at their current position when credentials expire, while writers cannot switch credentials in the middle of a native multipart upload. 2. **Lance: native callback.** Lance 0.39.0 already has `StorageOptionsProvider.fetchStorageOptions()`: when the storage options carry `expires_at_millis`, the native object store calls the Java provider before they expire. Paimon could implement it on top of `RESTTokenRefresher`. Today only the dataset APIs (`Dataset`, `OpenDatasetBuilder`, `ReadOptions`) accept a provider, not the file-level `LanceFileReader.open` / `LanceFileWriter.open` that Paimon uses, so this needs that support in Lance first. 3. **Vortex and Jindo.** Neither exposes a credential callback to Java today, so they need a similar hook upstream. For Jindo, pointing `provider.endpoint` at the REST server does not work because its token endpoint requires the REST catalog authentication, and serving tokens from a local HTTP endpoint inside the JVM would expose them to other local processes. I suggest starting with step 1, since it only touches Paimon, and following up on Lance with step 2. ### Anything else? This came up while working on #10178 and is left out of that PR, which covers the Java OSS SDK. Related: #10522. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
