singhpk234 commented on code in PR #15696:
URL: https://github.com/apache/iceberg/pull/15696#discussion_r2967164251
##########
gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSFileIO.java:
##########
@@ -199,13 +211,67 @@ private Map<String, PrefixedStorage> storageByPrefix() {
storageSupplier));
});
this.storageByPrefix = localStorageByPrefix;
+ scheduleCredentialRefresh();
}
}
}
return storageByPrefix;
}
+ private void scheduleCredentialRefresh() {
+ storageCredentials.stream()
+ .map(
+ storageCredential ->
+
storageCredential.config().get(GCPProperties.GCS_OAUTH2_TOKEN_EXPIRES_AT))
+ .filter(Objects::nonNull)
+ .map(expiresAtString ->
Instant.ofEpochMilli(Long.parseLong(expiresAtString)))
+ .min(Comparator.naturalOrder())
+ .ifPresent(
+ expiresAt -> {
+ Instant prefetchAt = expiresAt.minus(5, ChronoUnit.MINUTES);
+ long delay = Duration.between(Instant.now(),
prefetchAt).toMillis();
+ this.refreshFuture =
+ executorService()
+ .schedule(this::refreshStorageCredentials, delay,
TimeUnit.MILLISECONDS);
+ });
+ }
+
+ private void refreshStorageCredentials() {
+ if (isResourceClosed.get()) {
+ return;
+ }
+
+ try (OAuth2RefreshCredentialsHandler handler =
+ OAuth2RefreshCredentialsHandler.create(properties)) {
Review Comment:
> Typically, we're talking about possibly one invocation per hour (most
queries will likely have none because the run within the timeout)
If we create the handler lazily, we can achieve the same.
> holding all of the http client thread pools and other resources is the
more wasteful approach
should we trim down the http client pool to be way less than ~100 in the
first place ? we technically don't need the whole HTTP Client pool (~100) for
OAuth2RefreshCredsHandler (though i think pools connections might be lazily
made) ? its gonna be mostly call to rest server by picking a connection from
the pool .
My main suggestion was we are creating burst of connections / threads which
we are discarding anyways after the credentials call completes, hence i though
it was an expensive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
