[
https://issues.apache.org/jira/browse/HADOOP-19862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18090721#comment-18090721
]
ASF GitHub Bot commented on HADOOP-19862:
-----------------------------------------
konstantinb commented on code in PR #8426:
URL: https://github.com/apache/hadoop/pull/8426#discussion_r3455274073
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ASharedThreadPoolEnabled.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import org.assertj.core.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+
+import static
org.apache.hadoop.fs.s3a.Constants.AWS_S3_ASYNC_CLIENT_SHARED_THREADPOOL_ENABLED;
+import static
org.apache.hadoop.fs.s3a.Constants.AWS_S3_ASYNC_CLIENT_SHARED_THREADPOOL_SIZE;
+import static
org.apache.hadoop.fs.s3a.Constants.AWS_S3_CLIENT_SHARED_THREADPOOL_ENABLED;
+import static
org.apache.hadoop.fs.s3a.Constants.AWS_S3_CLIENT_SHARED_THREADPOOL_SIZE;
+
+/**
+ * Integration test demonstrating that the shared scheduled thread pool bounds
+ * thread growth across many S3AFileSystem instances.
+ * <p>
+ * Background (HADOOP-19862): each S3AFileSystem builds its own AWS SDK
clients,
+ * and each client otherwise creates a 5-thread sdk-ScheduledExecutor pool, so
+ * creating many instances grows threads without bound. With the shared pool
+ * enabled, all clients of a type reuse a single pool, so the thread count
stays
+ * bounded regardless of how many instances are created.
+ * <p>
+ * This test proves the fix direction (pool enabled, threads bounded);
+ * {@link ITestS3ASharedThreadPoolDisabled} is the control proving the opposite
+ * (pool disabled, threads grow). The two must be separate classes: the holders
+ * are private static final and memoize on first use, so the first
configuration
+ * to reach them wins for the JVM. They stay isolated because hadoop-aws runs
+ * tests with reuseForks=false (a fresh JVM per test class); do not merge them
+ * into one class.
+ * <p>
+ * Exact thread counts depend on AWS SDK internals, so the assertion is on the
+ * bound (not exceeding the pool sizes), not an exact number.
+ */
+public class ITestS3ASharedThreadPoolEnabled extends AbstractS3ATestBase {
Review Comment:
@steveloughran thank you for your review and additional comments
I went back and forth on this one. My current conclusion: teardown doesn't
need to touch the pool, so I've left it out and documented the constraint —
happy to add a hook if you'd prefer.
On resources, it cleans up after itself: the threads are daemon, and it sets
`allowCoreThreadTimeOut(true)` with a 60s keepalive, so idle threads time out
and the pool is reaped at JVM exit — it won't accumulate or hold the JVM open.
On isolation, a reset is the only lever — the holder is a JVM-global static
that memoizes on first use and then ignores later config. Under
`reuseForks=false` that reset is a no-op (each class already runs in its own
fresh JVM), and under fork reuse it'd be worse: `shutdownNow()` on the shared
pool would reject in-flight scheduled work for any other filesystem still using
it. So the enabled/disabled cases are split into two classes, each in its own
fork (which is what provides the isolation), and I've noted that in the
javadocs.
If you'd still like an explicit test-only shutdown hook regardless, I'm glad
to add one.
> S3A: Thread leak from AWS SDK v2 ScheduledExecutorService
> ---------------------------------------------------------
>
> Key: HADOOP-19862
> URL: https://issues.apache.org/jira/browse/HADOOP-19862
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Konstantin Bereznyakov
> Priority: Major
> Labels: pull-request-available
>
> AWS SDK v2 S3 clients create internal ScheduledExecutorService instances that
> accumulate over time, causing unbounded thread growth. Thread dumps show
> thousands of sdk-ScheduledExecutor-* threads in processes that create
> multiple S3AFileSystem instances.
> Environment
> - Hadoop 3.4.x with AWS SDK v2
> - Not observed in Hadoop 3.3.x (AWS SDK v1)
> Observed Behavior
> Thread dump comparison:
> Hadoop 3.3.x (AWS SDK v1): Normal thread count
> Hadoop 3.4.x (AWS SDK v2): 1600+ "sdk-ScheduledExecutor-*" threads
> Thread pattern:
> "sdk-ScheduledExecutor-0-0" daemon prio=5 waiting
> "sdk-ScheduledExecutor-0-1" daemon prio=5 waiting
> ...
> "sdk-ScheduledExecutor-0-4" daemon prio=5 waiting
> "sdk-ScheduledExecutor-1-0" daemon prio=5 waiting
> ...
> Root Cause
> AWS SDK v2's SdkDefaultClientBuilder creates a ScheduledThreadPoolExecutor
> with 5 threads per client when no executor is explicitly provided
> (https://github.com/aws/aws-sdk-java-v2/issues/1690):
> Executors.newScheduledThreadPool(5,
> new
> ThreadFactoryBuilder().threadNamePrefix("sdk-ScheduledExecutor").build())
> These threads are used for retry scheduling, timeout handling, and
> credential refresh.
> Contributing Factors
> 1. AbstractFileSystem has no caching
> Unlike FileSystem.get() which uses CACHE.get(uri, conf),
> AbstractFileSystem.get() always creates new instances:
> // AbstractFileSystem.java:263-266
> public static AbstractFileSystem get(final URI uri, final Configuration
> conf) {
> return createFileSystem(uri, conf); // NO CACHING
> }
> Each FileContext.getFileContext() call with an S3 URI creates:
> - New AbstractFileSystem (S3A)
> - New S3AFileSystem
> - New S3Client
> - 5 new sdk-ScheduledExecutor threads
> 2. S3Client threads not released on close
> As documented in https://github.com/aws/aws-sdk-java-v2/issues/1690:
> "When using cached, ephemeral clients, I can see that the scheduled thread
> pool will at times be leaked when the aws client is evicted"
> 3. Multiple client types affected
> S3A creates multiple AWS SDK clients:
> - S3Client (sync)
> - S3AsyncClient
> - STS client (for delegation tokens)
> - KMS client (for encryption)
> Each client instance creates its own 5-thread pool.
> Impact
> - Unbounded thread growth in any process using S3A
> - Resource exhaustion leading to OOM or system instability
> - Particularly affects YARN NodeManager, Spark drivers/executors, and other
> services that create many filesystem instances
> Related
> - https://github.com/aws/aws-sdk-java-v2/issues/1690 - SDK issue
> documenting the problem
> - https://github.com/aws/aws-sdk-java-v2/pull/4002 - SDK fix allowing
> shared executor configuration
> - HADOOP-19624 - Similar thread leak in ABFS (AbfsClientThrottlingAnalyzer)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
