Arnaud Nauwynck created MAPREDUCE-7465:
------------------------------------------
Summary: performance problem in FileOutputCommiter for big list
processed by single thread
Key: MAPREDUCE-7465
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7465
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: performance
Affects Versions: 3.3.6, 3.3.4, 3.3.3, 3.3.5, 3.2.4, 3.3.2, 3.2.3
Reporter: Arnaud Nauwynck
when commiting a big hadoop job (for example via Spark) having many partitions,
the class FileOutputCommiter process thousands of dirs/files to rename with a
single Thread. This is performance issue, caused by lot of waits on FileStystem
storage operations.
Notice that sub-class instances of FileOutputCommiter are supposed to be
created at runtime dependending of a configurable property
([https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PathOutputCommitterFactory.java|PathOutputCommitterFactory.java]).
But for example in Parquet + Spark, this is buggy and can not be changed at
runtime.
There is an ongoing Jira and PR to fix it in Parquet + Spark:
[https://issues.apache.org/jira/browse/PARQUET-2416|https://issues.apache.org/jira/browse/PARQUET-2416]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
