sunchao commented on code in PR #6371:
URL: https://github.com/apache/datafusion-comet/pull/6371#discussion_r4129199755
##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -537,7 +538,12 @@ object Utils extends CometTypeShim with Logging {
if (valueVector.getField.getDictionary != null)
Some(a.getDictionaryProvider)
else None
- (getFieldVector(valueVector, "serialize"), provider)
+ val fieldVector = valueVector match {
+ case v: LargeVarCharVector => narrowOffsets(v,
ArrowType.Utf8.INSTANCE)
Review Comment:
[P2] Could we release previously created copies when conversion of a later
column fails? With two large-offset columns, the first can successfully
allocate its narrowed copy before the second triggers the new oversized-payload
exception. The enclosing `map` then abandons the first copy before
`serializeBatches` creates its root, so `root.clear()` cannot release it.
Closing the input batch only releases the original vectors. Each such failure
therefore leaks the earlier columns' copied off-heap buffers. Track newly
materialized vectors and close them when batch assembly fails, while preserving
caller-owned vectors.
Evidence: A bounded probe used the exact helper bodies with Arrow 18.3.0:
column 0 contained `small`, and column 1 used the PR's existing `Int.MaxValue +
1L` offset fixture without allocating a large payload. Each failed call
retained 24 temporary bytes. After three calls and closing every input vector,
the allocator still reported 72 bytes. Replacing batch dispatch with the base
revision's implementation retained zero bytes. Reproduction and logs:
`/tmp/comet6371-current-review/run_probe.py`, `OffsetReviewProbe.log`, and
`BaseOffsetReviewProbe.log`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
