afterincomparableyum commented on code in PR #7124:
URL: https://github.com/apache/datafusion-comet/pull/7124#discussion_r4396969086
##########
spark/src/main/scala/org/apache/comet/vector/NativeUtil.scala:
##########
@@ -142,12 +142,22 @@ class NativeUtil extends AutoCloseable {
// Don't need to deallocate them here.
val arrowSchema = ArrowSchema.wrap(schemaAddrs(index))
val arrowArray = ArrowArray.wrap(arrayAddrs(index))
- Data.exportVector(
- allocator,
- getFieldVector(valueVector, "export"),
- provider,
- arrowArray,
- arrowSchema)
+ narrowLargeOffsets(valueVector) match {
+ case Some(narrow) =>
+ // A PyArrow UDF can return large_string or large_binary, which
the native side
+ // does not plan for. Export a 32-bit copy instead, as
serialization does. The
+ // export retains the copy's buffers, so the copy itself is
closed here and its
+ // memory is freed when native releases the array.
+ try Data.exportVector(allocator, narrow, provider, arrowArray,
arrowSchema)
+ finally narrow.close()
Review Comment:
Agreed, the copy isn't needed. exportBatch now passes these vectors through
and native casts them. I removed narrowLargeOffsets and added a native columnar
to row test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
