WangGuangxin opened a new pull request, #13009: URL: https://github.com/apache/gluten/pull/13009
## What changes are proposed in this pull request? Previously, Velox exported VARCHAR and VARBINARY columns using Arrow’s traditional Utf8 and Binary layouts. These layouts store all values in one contiguous data buffer and use signed 32-bit offsets. Consequently, a column fails to export when its aggregate payload exceeds approximately 2 GiB, even if every individual value is small. This PR adds an optional Arrow StringView export path for Velox-to-Java conversion. Utf8View and BinaryView use fixed-size descriptors that can reference multiple variadic data buffers, removing the requirement that an entire column fit in one contiguous buffer smaller than 2 GiB. The change includes: Exporting Velox strings as Utf8View/BinaryView through ColumnarBatches.load when supported. Adding Java accessors for ViewVarCharVector and ViewVarBinaryVector. Preserving View types during Arrow schema conversion. Preserving variadic-buffer metadata during load → offload → load round trips. Falling back to traditional Utf8/Binary when the Arrow runtime does not support StringView. Keeping shuffle, serialization, and writer paths unchanged to limit compatibility impact. This removes the approximately 2 GiB aggregate payload limit for a string or binary column in this conversion path. It does not support an individual value larger than approximately 2 GiB, and other paths that still use traditional Utf8/Binary retain the original limit. ### Why did the previous path have a 2 GiB limit? Traditional Arrow Utf8 and Binary arrays store all values in one contiguous data buffer and use signed 32-bit offsets: validity buffer offset buffer: [0, len0, len0 + len1, ...] data buffer: all values stored contiguously The last offset represents the column’s total payload size and cannot exceed Integer.MAX_VALUE (2,147,483,647). Therefore, conversion can fail when the aggregate payload of one column exceeds approximately 2 GiB, even if every individual value is small. ### How does the new path bypass this limit? Arrow Utf8View and BinaryView use fixed-size 16-byte descriptors. Short values can be stored inline, while longer values reference one of multiple variadic data buffers using a buffer index and offset. This removes the requirement that all values in a column be stored in one contiguous buffer smaller than 2 GiB. The aggregate payload can therefore exceed 2 GiB by being distributed across multiple backing buffers. ### What cases are still not addressed? A single string or binary value larger than approximately 2 GiB is still unsupported because its length is represented using a 32-bit integer, and Java arrays and Spark UTF8String have similar limits. Paths that still use traditional Utf8/Binary, such as shuffle, serialization, and writer paths, retain the original aggregate 2 GiB limit. Environments without Arrow StringView support, such as the default Arrow 15/JDK 8 configuration, fall back to traditional Utf8/Binary. Normal memory and allocator limits still apply. StringView avoids the contiguous-buffer offset limit; it does not provide unlimited capacity. ## How was this patch tested? Added tests covering: Velox export using Utf8View and BinaryView. Java access through ArrowWritableColumnVector and ArrowColumnVector. Null, inline, and out-of-line View values. View variadic-buffer metadata. Velox → Arrow load and load → offload → load round trips. Arrow 15 fallback compatibility. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: GPT-5.6 Sol -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
