westonpace commented on a change in pull request #9644:
URL: https://github.com/apache/arrow/pull/9644#discussion_r601756334
##########
File path: cpp/src/arrow/csv/reader.cc
##########
@@ -199,6 +199,19 @@ class SerialBlockReader : public BlockReader {
return MakeTransformedIterator(std::move(buffer_iterator),
block_reader_fn);
}
+ static AsyncGenerator<CSVBlock> MakeAsyncIterator(
+ AsyncGenerator<std::shared_ptr<Buffer>> buffer_generator,
+ std::unique_ptr<Chunker> chunker, std::shared_ptr<Buffer> first_buffer) {
+ auto block_reader =
+ std::make_shared<SerialBlockReader>(std::move(chunker), first_buffer);
+ // Wrap shared pointer in callable
+ Transformer<std::shared_ptr<Buffer>, CSVBlock> block_reader_fn =
+ [block_reader](std::shared_ptr<Buffer> next) {
+ return (*block_reader)(std::move(next));
+ };
+ return MakeAsyncGenerator(std::move(buffer_generator), block_reader_fn);
Review comment:
In my early days implementing AsyncGenerator I thought all operators
were going to be variations of Transformer<T, V> and so this would be the
de-facto way of creating new operators. I was aiming for something equivalent
to RxJS' lift/pipe but this idea failed in practice.
I've renamed to `MakeTransformedGenerator` and should probably investigate
removing this entirely (in favor of `MakeMappedGenerator` or custom generators
if needed) at some point in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
