Ivan Sadikov created SPARK-62048:
------------------------------------

             Summary: MemoryStore.putIteratorAsBytes leaks the compression 
stream when unrolling a block fails
                 Key: SPARK-62048
                 URL: https://issues.apache.org/jira/browse/SPARK-62048
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 4.2.2, 5.0.0
            Reporter: Ivan Sadikov


MemoryStore.putIteratorAsBytes creates a SerializedValuesHolder, whose 
constructor opens a serialization stream wrapped by 
SerializerManager.wrapForCompression. The stream is closed in only two cases:
- the block is fully unrolled: SerializedValuesHolder.getBuilder() closes it;
- the block is partially unrolled: the stream is handed to a 
PartiallySerializedBlock, which closes it in discard() or 
finishWritingToStream().

putIterator has no failure handling. If values.next() or 
valuesHolder.storeValue() throws, the exception escapes putIteratorAsBytes and 
nothing closes the stream or releases the unrolled buffers.

With the default zstd codec, ZStdCompressionCodec returns a 
ZstdOutputStreamNoFinalizer. Its native compression context and buffers are 
released only by close(), not by GC, so the executor leaks one native zstd 
context each time a task fails while caching a block as serialized bytes (for 
example MEMORY_ONLY_SER or MEMORY_AND_DISK_SER). Common non-fatal triggers:
- an exception thrown by the RDD's input iterator (user code);
- NotSerializableException from the Java serializer;
- KryoException from the Kryo serializer.

Because the task fails but the executor survives, the native memory adds up 
across task retries and long-running executors and grows off-heap usage outside 
the JVM heap limits.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to