Julien Le Dem created PARQUET-91:
------------------------------------
Summary: stream through files when writing the _metadata file to
reduce memory usage
Key: PARQUET-91
URL: https://issues.apache.org/jira/browse/PARQUET-91
Project: Parquet
Issue Type: Improvement
Components: parquet-mr
Reporter: Julien Le Dem
Currently we load all the footers of the part files in memory in the committer
to write the _metadata file.
We could open the _metadata file first and then add the metadata for each part
file one by one. That would reduce the required memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
