Huanli Wang created SPARK-53549:
-----------------------------------
Summary: Fix the list state memory leak issue due to an unclosed
Arrow allocator
Key: SPARK-53549
URL: https://issues.apache.org/jira/browse/SPARK-53549
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 4.0.1, 4.0.0, 4.1.0
Reporter: Huanli Wang
We create a
[TransformWithStateInPySparkDeserializer|https://github.com/apache/spark/blob/7201e60bb3e898f65db5045ec293c650e14cfa95/sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala#L460-L464]
every time when we process a list state request. Inside
[TransformWithStateInPySparkDeserializer|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkDeserializer.scala#L40],
a child allocator is created from the *ArrowUtils.rootAllocator* when a new
deserializer instance is created. However, we neither close this child
allocator nor call rootAllocator.close, which leads to a memory leak in
TWSStateServer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
