singhpk234 commented on code in PR #15696:
URL: https://github.com/apache/iceberg/pull/15696#discussion_r2967164251
##########
gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSFileIO.java:
##########
@@ -199,13 +211,67 @@ private Map<String, PrefixedStorage> storageByPrefix() {
storageSupplier));
});
this.storageByPrefix = localStorageByPrefix;
+ scheduleCredentialRefresh();
}
}
}
return storageByPrefix;
}
+ private void scheduleCredentialRefresh() {
+ storageCredentials.stream()
+ .map(
+ storageCredential ->
+
storageCredential.config().get(GCPProperties.GCS_OAUTH2_TOKEN_EXPIRES_AT))
+ .filter(Objects::nonNull)
+ .map(expiresAtString ->
Instant.ofEpochMilli(Long.parseLong(expiresAtString)))
+ .min(Comparator.naturalOrder())
+ .ifPresent(
+ expiresAt -> {
+ Instant prefetchAt = expiresAt.minus(5, ChronoUnit.MINUTES);
+ long delay = Duration.between(Instant.now(),
prefetchAt).toMillis();
+ this.refreshFuture =
+ executorService()
+ .schedule(this::refreshStorageCredentials, delay,
TimeUnit.MILLISECONDS);
+ });
+ }
+
+ private void refreshStorageCredentials() {
+ if (isResourceClosed.get()) {
+ return;
+ }
+
+ try (OAuth2RefreshCredentialsHandler handler =
+ OAuth2RefreshCredentialsHandler.create(properties)) {
Review Comment:
> Typically, we're talking about possibly one invocation per hour (most
queries will likely have none because the run within the timeout)
If we create the handler lazily, we can achieve the same.
> holding all of the http client thread pools and other resources is the
more wasteful approach
should we trim down the http client pool to be 1 in the first place ? we
technically don't need the whole HTTP Client pool (~100) for
OAuth2RefreshCredsHandler ? its gonna be mostly single threaded.
My main suggestion was we are creating burst of connections / threads which
we are discarding anyways after the credentials call completes, hence was an
expensive thought.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
