kylebarron commented on code in PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#discussion_r4148226475
##########
arrow-array/src/ffi_stream.rs:
##########
@@ -533,6 +548,78 @@ impl RecordBatchReader for ArrowArrayStreamReader {
}
}
+/// An [`ArrayReader`] which imports arrays from an [`FFI_ArrowArrayStream`].
+///
+/// Unlike [`ArrowArrayStreamReader`], the stream's arrays may be of **any**
data type. This
+/// supports reading a stream of generic arrays that may not represent record
batches.
Review Comment:
This is the entire distinction and is important:
- The `ArrowArrayStream` sends _arrays_ not record batches through FFI.
- The _`RecordBatch`_ based stream reader **requires that the data type is
`Struct`**, but then allows the _children_ of the StructArray to be anything.
I.e. the top-level `StructArray` in the `ArrowArrayStream` is automatically
unpacked to a `RecordBatch`. This fails for any stream where the data type is
not `Struct`.
- The _`Array`_ based stream reader lets the item passed through the
`ArrowArrayStream` be something _other_ than `Struct`
Currently arrow-rs only defines the RecordBatch-based stream reader, so it
**only** supports sharing `Struct`-based arrays via FFI.
There are some use cases where users want to share streams of bare arrays
that are not `RecordBatch`es.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
