peterxcli opened a new issue, #5299: URL: https://github.com/apache/datafusion-comet/issues/5299
## What is the problem the feature request solves? `ArrowWriter.writeColNoNull` still copies fixed-width columns one value at a time with `setValueUnsafe`. This avoids Arrow's per-value capacity check, but retains a getter call and setter call for every value. A bulk copy into the Arrow data buffer could remove that per-row dispatch for eligible Spark column vectors. This was identified during review of #5046. ## Describe the potential solution Investigate a specialized no-null path for fixed-width types whose Spark and Arrow physical layouts match: - copy the contiguous value region into `BaseFixedWidthVector.getDataBuffer`; - initialize the destination validity bitmap in bulk; - preserve the existing per-value path for unsupported vectors and layouts; - handle sliced input offsets, dictionaries, endianness, and both on-heap and off-heap inputs; - benchmark the specialized path across representative types and batch sizes before adopting it. There are API constraints to resolve first: - `OffHeapColumnVector.valuesNativeAddress` exposes the values address but is annotated `@VisibleForTesting`; - `OnHeapColumnVector` keeps its primitive arrays private, while its public bulk getters allocate new arrays; - `ColumnarArray` keeps its backing vector and offset private, so the current writer API does not expose the information needed for a direct copy; - booleans are byte-per-value in Spark but bit-packed in Arrow, and decimals/intervals do not generally share a directly copyable layout. Avoid reflection or private-field access that would be brittle across supported Spark versions. The issue should only proceed if benchmarks show that a stable source-access approach beats the current loop. ## Additional context - Parent optimization: #5046 - Review summary: https://github.com/apache/datafusion-comet/pull/5046#pullrequestreview-4877706912 - Inline suggestion: https://github.com/apache/datafusion-comet/pull/5046#discussion_r3731420254 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
