praveenc7 opened a new pull request, #18262:
URL: https://github.com/apache/pinot/pull/18262
## Description
Adds the foundation types needed to represent MSE data as Apache Arrow
VectorSchemaRoots, behind a feature flag that is off by default.
Added foundation classes
- ArrowDataBlock — DataBlock implementation over Arrow VectorSchemaRoot
- ArrowBlock — MseBlock.Data wrapper with reference-counted lifecycle
- ArrowBlockConverter — converts RowHeapDataBlock/SerializedDataBlock →
ArrowBlock column-by-column, with a fast path for ColumnarDataBlock that copies
directly from the backing buffer
- ArrowBuffers — manages the Arrow RootAllocator lifecycle; not a
singleton, created and injected explicitly
Currently with this PR
- No operator creates ArrowBlocks
- No operator consumes ArrowBlocks differently
- MailboxReceiveOperator does not convert incoming blocks
- HashJoinOperator is unchanged
- Wire format is unchanged — if an ArrowBlock does reach
GrpcSendingMailbox (e.g., via a test), it falls back through asRowHeap() →
legacy protobuf, same as before
## Testing done
Unit test added
## Future PRs
- Arrow-native HashJoinOperator with Arrow-aware KeySelector
hashers/comparators
- Arrow IPC serialization in GrpcSendingMailbox (metadata field selects
format; same gRPC transport)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
