[
https://issues.apache.org/jira/browse/SPARK-35181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407065#comment-17407065
]
Dongjoon Hyun commented on SPARK-35181:
---------------------------------------
[~angerszhuuu], there is no problem if you upgrade your zstd library and
backport all relevant changes.
In addition to the code change, if you are on K8s, you need to tune your Spark
jobs' configuration based on your memory usage.
> Use zstd for spark.io.compression.codec by default
> --------------------------------------------------
>
> Key: SPARK-35181
> URL: https://issues.apache.org/jira/browse/SPARK-35181
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.2.0
> Reporter: Dongjoon Hyun
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
