talatuyarer commented on code in PR #16848:
URL: https://github.com/apache/iceberg/pull/16848#discussion_r4006783571
##########
gcp/src/main/java/org/apache/iceberg/gcp/auth/GoogleAuthManager.java:
##########
@@ -66,7 +67,7 @@ public class GoogleAuthManager implements AuthManager {
private final String name;
private GoogleCredentials credentials;
- private boolean initialized = false;
+ private volatile boolean initialized = false;
Review Comment:
The locking looks right to me, nice. One thought though: the other places we
do this in the codebase (OAuth2Manager, VendedCredentialsProvider,
PrefixedS3Client) just make the field itself volatile and null-check it,
instead of keeping a separate flag. Would you mind doing the same here? Then
`initialized` goes away and the `checkState(credentials != null)` in
`catalogSession` isn't needed anymore either.
##########
gcp/src/main/java/org/apache/iceberg/gcp/auth/GoogleAuthManager.java:
##########
@@ -81,44 +82,68 @@ private void initialize(Map<String, String> properties) {
return;
}
- String credentialsPath = properties.get(GCP_CREDENTIALS_PATH_PROPERTY);
- String credentialsJson = properties.get(GCP_CREDENTIALS_JSON_PROPERTY);
- boolean useCredentialsPath = credentialsPath != null &&
!credentialsPath.isEmpty();
- boolean useCredentialsJson = credentialsJson != null &&
!credentialsJson.isEmpty();
- if (useCredentialsPath && useCredentialsJson) {
- throw new IllegalArgumentException(
- String.format(
- "Cannot specify both %s and %s",
- GCP_CREDENTIALS_PATH_PROPERTY, GCP_CREDENTIALS_JSON_PROPERTY));
+ synchronized (this) {
+ if (initialized) {
+ return;
+ }
+
+ String credentialsPath = properties.get(GCP_CREDENTIALS_PATH_PROPERTY);
+ String credentialsJson = properties.get(GCP_CREDENTIALS_JSON_PROPERTY);
+ boolean useCredentialsPath = credentialsPath != null &&
!credentialsPath.isEmpty();
+ boolean useCredentialsJson = credentialsJson != null &&
!credentialsJson.isEmpty();
+ if (useCredentialsPath && useCredentialsJson) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Cannot specify both %s and %s",
+ GCP_CREDENTIALS_PATH_PROPERTY, GCP_CREDENTIALS_JSON_PROPERTY));
+ }
+
+ String scopesString = properties.getOrDefault(GCP_SCOPES_PROPERTY,
DEFAULT_SCOPES);
+
+ try {
+ this.credentials =
+ loadCredentials(
+ useCredentialsPath,
+ credentialsPath,
+ useCredentialsJson,
+ credentialsJson,
+ scopesString);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to load Google credentials", e);
+ }
+
+ this.initialized = true;
}
+ }
- String scopesString = properties.getOrDefault(GCP_SCOPES_PROPERTY,
DEFAULT_SCOPES);
+ @VisibleForTesting
+ GoogleCredentials loadCredentials(
Review Comment:
I see why you pulled this out, the static mock doesn't work across threads
so there's no other way to stub it from the test. That's fine, but can we drop
the two booleans? They're just `!Strings.isNullOrEmpty(...)` of the strings
right next to them, and we generally try not to have boolean params on
non-private methods. `loadCredentials(credentialsPath, credentialsJson,
scopesString)` and compute the flags inside would be cleaner.
##########
gcp/src/test/java/org/apache/iceberg/gcp/auth/TestGoogleAuthManager.java:
##########
@@ -221,6 +227,42 @@ public void initializationOccursOnlyOnce() {
mockedStaticCredentials.verify(GoogleCredentials::getApplicationDefault,
times(1));
}
+ @Test
+ public void concurrentInitialization() throws Exception {
+ int numThreads = 10;
Review Comment:
nit: maybe pull the thread count and the timeout out as constants.
##########
gcp/src/test/java/org/apache/iceberg/gcp/auth/TestGoogleAuthManager.java:
##########
@@ -221,6 +227,42 @@ public void initializationOccursOnlyOnce() {
mockedStaticCredentials.verify(GoogleCredentials::getApplicationDefault,
times(1));
}
+ @Test
+ public void concurrentInitialization() throws Exception {
+ int numThreads = 10;
+ ExecutorService executorService = Executors.newFixedThreadPool(numThreads);
+ CountDownLatch startLatch = new CountDownLatch(1);
+ CountDownLatch finishLatch = new CountDownLatch(numThreads);
+
+ GoogleAuthManager spyManager = spy(authManager);
+ doReturn(credentials)
+ .when(spyManager)
+ .loadCredentials(anyBoolean(), any(), anyBoolean(), any(), any());
+
+ AtomicInteger successfulInitializations = new AtomicInteger(0);
+ for (int i = 0; i < numThreads; i++) {
+ executorService.submit(
+ () -> {
+ try {
+ startLatch.await();
+ spyManager.catalogSession(restClient, Collections.emptyMap());
+ successfulInitializations.incrementAndGet();
+ } catch (Exception e) {
+ // ignore
+ } finally {
+ finishLatch.countDown();
+ }
+ });
+ }
+
+ startLatch.countDown();
+ finishLatch.await(10, TimeUnit.SECONDS);
+ executorService.shutdown();
Review Comment:
This should be in a `finally`, otherwise the pool leaks when the assertion
fails. `shutdownNow()` there is fine.
##########
gcp/src/test/java/org/apache/iceberg/gcp/auth/TestGoogleAuthManager.java:
##########
@@ -221,6 +227,42 @@ public void initializationOccursOnlyOnce() {
mockedStaticCredentials.verify(GoogleCredentials::getApplicationDefault,
times(1));
}
+ @Test
+ public void concurrentInitialization() throws Exception {
+ int numThreads = 10;
+ ExecutorService executorService = Executors.newFixedThreadPool(numThreads);
+ CountDownLatch startLatch = new CountDownLatch(1);
+ CountDownLatch finishLatch = new CountDownLatch(numThreads);
+
+ GoogleAuthManager spyManager = spy(authManager);
+ doReturn(credentials)
+ .when(spyManager)
+ .loadCredentials(anyBoolean(), any(), anyBoolean(), any(), any());
+
+ AtomicInteger successfulInitializations = new AtomicInteger(0);
+ for (int i = 0; i < numThreads; i++) {
+ executorService.submit(
+ () -> {
+ try {
+ startLatch.await();
+ spyManager.catalogSession(restClient, Collections.emptyMap());
+ successfulInitializations.incrementAndGet();
+ } catch (Exception e) {
Review Comment:
Please don't swallow these. If one of the threads blows up you'll just see
`9 != 10` at the bottom with no idea why. Either stash the exceptions in a list
and assert it's empty, or submit `Callable`s and `get()` the futures so it
propagates.
##########
gcp/src/test/java/org/apache/iceberg/gcp/auth/TestGoogleAuthManager.java:
##########
@@ -221,6 +227,42 @@ public void initializationOccursOnlyOnce() {
mockedStaticCredentials.verify(GoogleCredentials::getApplicationDefault,
times(1));
}
+ @Test
+ public void concurrentInitialization() throws Exception {
+ int numThreads = 10;
+ ExecutorService executorService = Executors.newFixedThreadPool(numThreads);
+ CountDownLatch startLatch = new CountDownLatch(1);
+ CountDownLatch finishLatch = new CountDownLatch(numThreads);
+
+ GoogleAuthManager spyManager = spy(authManager);
Review Comment:
nit: `initializationOccursOnlyOnce` already covers the single-init part with
the static mock, so the thing this test adds is the race. Might be worth a
one-line comment saying that, and that it's a best-effort regression check
since the unsynchronized window is tiny. Otherwise someone might see it flake
once and think the fix is broken.
##########
gcp/src/main/java/org/apache/iceberg/gcp/auth/GoogleAuthManager.java:
##########
@@ -81,44 +82,68 @@ private void initialize(Map<String, String> properties) {
return;
}
- String credentialsPath = properties.get(GCP_CREDENTIALS_PATH_PROPERTY);
- String credentialsJson = properties.get(GCP_CREDENTIALS_JSON_PROPERTY);
- boolean useCredentialsPath = credentialsPath != null &&
!credentialsPath.isEmpty();
- boolean useCredentialsJson = credentialsJson != null &&
!credentialsJson.isEmpty();
- if (useCredentialsPath && useCredentialsJson) {
- throw new IllegalArgumentException(
- String.format(
- "Cannot specify both %s and %s",
- GCP_CREDENTIALS_PATH_PROPERTY, GCP_CREDENTIALS_JSON_PROPERTY));
+ synchronized (this) {
+ if (initialized) {
+ return;
+ }
+
+ String credentialsPath = properties.get(GCP_CREDENTIALS_PATH_PROPERTY);
+ String credentialsJson = properties.get(GCP_CREDENTIALS_JSON_PROPERTY);
+ boolean useCredentialsPath = credentialsPath != null &&
!credentialsPath.isEmpty();
+ boolean useCredentialsJson = credentialsJson != null &&
!credentialsJson.isEmpty();
+ if (useCredentialsPath && useCredentialsJson) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Cannot specify both %s and %s",
+ GCP_CREDENTIALS_PATH_PROPERTY, GCP_CREDENTIALS_JSON_PROPERTY));
+ }
+
+ String scopesString = properties.getOrDefault(GCP_SCOPES_PROPERTY,
DEFAULT_SCOPES);
+
+ try {
+ this.credentials =
+ loadCredentials(
+ useCredentialsPath,
+ credentialsPath,
+ useCredentialsJson,
+ credentialsJson,
+ scopesString);
+ } catch (IOException e) {
Review Comment:
Just checking my understanding: if loading throws, `initialized` stays false
and the next call retries. That's the same as before, so no change needed.
Maybe a short comment on the field so it's obvious that's intentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
