[ https://issues.apache.org/jira/browse/SPARK-20406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-20406. ------------------------------- Resolution: Duplicate Search JIRA first please > Upgrade parquet to 1.9 > ---------------------- > > Key: SPARK-20406 > URL: https://issues.apache.org/jira/browse/SPARK-20406 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.1.0 > Reporter: Kaspar Sakmann > Labels: maven > > Please upgrade parquet to 1.9. > with 500G memory 48 CPUs and a dataset about 1G in size > using Parquet 1.8 causes warnings: > "Total allocation exceeds 95.00% (2,147,483,647 bytes) of heap memory > Scaling row group sizes to 34.04% for 47 writers" > The issue appears to have been fixed with parquet 1.9, see > https://issues.apache.org/jira/browse/PARQUET-340 -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org