[
https://issues.apache.org/jira/browse/SPARK-50235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-50235:
-----------------------------------
Labels: pull-request-available (was: )
> Clean up ColumnVector resource after processing all rows in ColumnarToRowExec
> -----------------------------------------------------------------------------
>
> Key: SPARK-50235
> URL: https://issues.apache.org/jira/browse/SPARK-50235
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 3.4.4, 3.5.3
> Reporter: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> Currently we only assign null to ColumnarBatch object but it doesn't release
> the resources hold by the vectors in the batch. For OnHeapColumnVector, the
> Java arrays may be automatically collected by JVM, but for
> OffHeapColumnVector, the allocated off-heap memory will be leaked.
> For custom ColumnVector implementations like Arrow-based, it also possibly
> causes issues on memory safety if the underlying buffers are reused across
> batches. Because when ColumnarToRowExec begins to fill values for next batch,
> the arrays in previous batch are still hold.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
