[ https://issues.apache.org/jira/browse/SPARK-41952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-41952: ------------------------------------ Assignee: (was: Apache Spark) > Upgrade Parquet to fix off-heap memory leaks in Zstd codec > ---------------------------------------------------------- > > Key: SPARK-41952 > URL: https://issues.apache.org/jira/browse/SPARK-41952 > Project: Spark > Issue Type: Bug > Components: Input/Output > Affects Versions: 3.1.3, 3.3.1, 3.2.3 > Reporter: Alexey Kudinkin > Priority: Critical > > Recently, native memory leak have been discovered in Parquet in conjunction > of it using Zstd decompressor from luben/zstd-jni library (PARQUET-2160). > This is very problematic to a point where we can't use Parquet w/ Zstd due to > pervasive OOMs taking down our executors and disrupting our jobs. > Luckily fix addressing this had already landed in Parquet: > [https://github.com/apache/parquet-mr/pull/982] > > Now, we just need to > # Updated version of Parquet is released in a timely manner > # Spark is upgraded onto this new version in the upcoming release > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org