Steve Loughran created HADOOP-19743:
---------------------------------------
Summary: S3A: ITestS3ACommitterMRJob MR process OOM on java17
Key: HADOOP-19743
URL: https://issues.apache.org/jira/browse/HADOOP-19743
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.5.0
Reporter: Steve Loughran
Assignee: Steve Loughran
MR AppMaster launched by S3ACommitterMRJob can OOM on java17 while launching
because the heap size is set for all forked processes to 128M.
{code}
ava.lang.OutOfMemoryError: Java heap space
at
java.base/java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:75)
at
org.apache.hadoop.fs.statistics.BufferedIOStatisticsOutputStream.<init>(BufferedIOStatisticsOutputStream.java:78)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:629)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:660)
{code}
Fix: double it. Why not larger? increases memory footprint and in parallel test
runs with many junit threads running there's a lot of load on memory
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
