yongster commented on issue #10800: URL: https://github.com/apache/arrow-rs/issues/10800#issuecomment-5461849577
Thanks @alamb and @tustvold for the guidance. The problem I was trying to solve is Spark shuffle size: after a sparse `filter` / `take` on `Utf8View` / `BinaryView`, unused data buffers are still serialized over IPC. I agree we should not change `filter` / `take` / coalescer heuristics for that — those kernels should stay zero-copy by default. The existing `StringViewArray::gc()` (and `garbage_collect_dictionary` for dictionaries) already cover compacting at the shuffle / IPC write boundary. Spark can call those before sending, so no kernel change is needed here. Closing this. Happy to follow up in a separate issue if a convenience compact helper later proves useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
