[
https://issues.apache.org/jira/browse/SPARK-39702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen updated SPARK-39702:
-------------------------------
Component/s: YARN
> Reduce memory overhead of TransportCipher$EncryptedMessage's byteRawChannel
> buffer
> ----------------------------------------------------------------------------------
>
> Key: SPARK-39702
> URL: https://issues.apache.org/jira/browse/SPARK-39702
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 3.0.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Major
>
> With Spark's encryption enabled ({{{}spark.network.crypto.enabled{}}} set to
> {{true}} and {{spark.network.crypto.saslFallback}} set to {{{}false{}}}), I
> ran into memory usage problems in the external shuffle service.
> This was caused by a problem that is very similar to SPARK-24801: eachÂ
> {{TransportCipher$EncryptedMessage}} eagerly initializes a buffer that is
> used during the encryption process. This buffer is only used once
> {{transferTo}} is called, but it is eagerly initialized in the
> {{EncryptedMessage}} constructor. This leads to high memory usage when there
> are many messages queued in an outgoing channel.
> One possible fix would be to mimic SPARK-24801 and make the initialization
> lazy. However, we can actually go one step further and share a single re-used
> buffer across multiple messages. This is safe because those messages are
> _already_ sharing a different buffer which is accessed in the same write
> paths.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
