singhpk234 commented on code in PR #15696:
URL: https://github.com/apache/iceberg/pull/15696#discussion_r2966028155
##########
gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSFileIO.java:
##########
@@ -199,13 +211,67 @@ private Map<String, PrefixedStorage> storageByPrefix() {
storageSupplier));
});
this.storageByPrefix = localStorageByPrefix;
+ scheduleCredentialRefresh();
Review Comment:
[optional] i wonder if its just about making sure before the obj is
serialized, we have set of credentials so that the executor when deserializes
the obj, it does not hit the server with bcz the creds in FileIO obj expired a
while ago ....
can we override the writeObj() (which JavaSerializer would call) which does
the creds call and then just make this call instead of the periodic refresh ?
I know there are rough edges for Kryo i was reading there is
`writeReplace()` which works for both
##########
gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSFileIO.java:
##########
@@ -199,13 +211,67 @@ private Map<String, PrefixedStorage> storageByPrefix() {
storageSupplier));
});
this.storageByPrefix = localStorageByPrefix;
+ scheduleCredentialRefresh();
}
}
}
return storageByPrefix;
}
+ private void scheduleCredentialRefresh() {
+ storageCredentials.stream()
+ .map(
+ storageCredential ->
+
storageCredential.config().get(GCPProperties.GCS_OAUTH2_TOKEN_EXPIRES_AT))
+ .filter(Objects::nonNull)
+ .map(expiresAtString ->
Instant.ofEpochMilli(Long.parseLong(expiresAtString)))
+ .min(Comparator.naturalOrder())
+ .ifPresent(
+ expiresAt -> {
+ Instant prefetchAt = expiresAt.minus(5, ChronoUnit.MINUTES);
+ long delay = Duration.between(Instant.now(),
prefetchAt).toMillis();
+ this.refreshFuture =
+ executorService()
+ .schedule(this::refreshStorageCredentials, delay,
TimeUnit.MILLISECONDS);
+ });
+ }
+
+ private void refreshStorageCredentials() {
+ if (isResourceClosed.get()) {
+ return;
+ }
+
+ try (OAuth2RefreshCredentialsHandler handler =
+ OAuth2RefreshCredentialsHandler.create(properties)) {
Review Comment:
creating a OAuth2RefreshCredentialsHandler per refresh call would be
expensive, it creates a http client and an AuthManager additionally, i wonder
if we can just have this done one (invalidated / created again if props change)
and just reuse that across refreshes ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
