[
https://issues.apache.org/jira/browse/ARROW-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17288568#comment-17288568
]
Weston Pace commented on ARROW-1009:
------------------------------------
I'm a little confused by the ticket I'm afraid. The only `StreamReader` I see
is `parquet/stream_reader.h` and that is focused on C++ `>>` streams. It
allows the user to read elements out of a parquet file one at a time
(presumably into some non-arrow format?)
There is also the `RecordBatchStreamReader` which allows for reading a source
one record batch at a time (instead of an entire table at a time). An
asynchronous mirror version of this API would make sense in the same way it
would make sense to have such an API for any I/O operation. The implementation
would probably come after an asynchronous table reader for the given format
(e.g. once an asynchronous table reader is done for parquet then one can tackle
an asynchronous stream reader).
There is StreamDecoder which allows processing buffers through observation. It
would have similar memory pressure to RecordBatchStreamReader if I understand
because the stream decoder will buffer up to a record batch and then deliver
it. It's also synchronous and there could be an asynchronous version (i.e.
`ipc::StreamDecoder::ConsumeAsync`).
Then there is flight mentioned in the comments. Flight uses
`MetadataRecordBatchReader`. It could be made asynchronous.
For the flight client/server I agree that there might not be a strong
motivation right now. Since it isn't yet used internally by Arrow (in the same
way the query/dataset APIs use the various readers internally) it isn't as
needed. I think it could still be a low priority improvement to keep I/O on
the IOContext and CPU on the CPU thread pool. Any consumers can do this
themselves but it would be a "nice to have" if Arrow took care of the queuing
for them.
Do we think a flight server might serve up hundreds/thousands of concurrent
downloads? For example, you might have this in video hosting (e.g. YouTube)
but there is a natural throttling in place there (no need to send data faster
than it is being displayed) that might not apply for data analytics.
[~wesm] Can you provide a few more details what the intent is for this issue?
I think the highest priority of all the things I mentioned is getting an
asynchronous record batch reader from parquet (will be needed for dataset
nested parallelism) and I think we are close given the prebuffering is already
pushing I/O to the IOContext.
> [C++] Create asynchronous version of StreamReader
> -------------------------------------------------
>
> Key: ARROW-1009
> URL: https://issues.apache.org/jira/browse/ARROW-1009
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Wes McKinney
> Priority: Major
> Fix For: 4.0.0
>
>
> the {{AsyncStreamReader}} would buffer the next record batch in a background
> thread, while emulating the current synchronous / blocking API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
