Rajesh Balamohan created HADOOP-18347:
-----------------------------------------
Summary: Restrict vectoredIO threadpool to reduce memory pressure
Key: HADOOP-18347
URL: https://issues.apache.org/jira/browse/HADOOP-18347
Project: Hadoop Common
Issue Type: Sub-task
Components: common, fs, fs/adl, fs/s3
Reporter: Rajesh Balamohan
https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L964-L967
Currently, it fetches all the ranges with unbounded threadpool. This will not
cause memory pressures with standard benchmarks like TPCDS. However, when large
number of ranges are present with large files, this could potentially spike up
memory usage of the task. Limiting the threadpool size could reduce the memory
usage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
