Alexey Kudinkin created SPARK-41952: ---------------------------------------
Summary: Upgrade Parquet to fix off-heap memory leaks in Zstd codec Key: SPARK-41952 URL: https://issues.apache.org/jira/browse/SPARK-41952 Project: Spark Issue Type: Bug Components: Input/Output Affects Versions: 3.2.3, 3.3.1, 3.1.3 Reporter: Alexey Kudinkin Recently, native memory leak have been discovered in Parquet in conjunction of it using Zstd decompressor from luben/zstd-jni library (PARQUET-2160). This is very problematic to a point where we can't use Parquet w/ Zstd due to pervasive OOMs taking down our executors and disrupting our jobs. Luckily fix addressing this had already landed in Parquet: [https://github.com/apache/parquet-mr/pull/982] Now, we just need to # Updated version of Parquet is released in a timely manner # Spark is upgraded onto this new version in the upcoming release -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org