kylebarron commented on code in PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#discussion_r4148226475
##########
arrow-array/src/ffi_stream.rs:
##########
@@ -533,6 +548,78 @@ impl RecordBatchReader for ArrowArrayStreamReader {
}
}
+/// An [`ArrayReader`] which imports arrays from an [`FFI_ArrowArrayStream`].
+///
+/// Unlike [`ArrowArrayStreamReader`], the stream's arrays may be of **any**
data type. This
+/// supports reading a stream of generic arrays that may not represent record
batches.
Review Comment:
This is the entire distinction and is important:
- The _`Array`_ based stream reader lets the item passed through the
`ArrowArrayStream` be something _other_ than `Struct`
- The _`RecordBatch`_ based stream reader **requires that the data type is
`Struct`**, but then allows the _children_ of the StructArray to be anything.
I.e. the top-level `StructArray` in the `ArrowArrayStream` is automatically
unpacked to a `RecordBatch`.
Currently arrow-rs only defines the latter, so it **only** supports sharing
`Struct`-based arrays via FFI.
There are some use cases where users want to share streams of bare arrays
that are not `RecordBatch`es.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
