lidavidm commented on a change in pull request #10061:
URL: https://github.com/apache/arrow/pull/10061#discussion_r616200811
##########
File path: cpp/src/arrow/dataset/file_csv.cc
##########
@@ -154,9 +180,26 @@ class CsvScanTask : public ScanTask {
source_(fragment->source()) {}
Result<RecordBatchIterator> Execute() override {
- ARROW_ASSIGN_OR_RAISE(auto reader,
- OpenReader(source_, *format_, options(),
options()->pool));
- return IteratorFromReader(std::move(reader));
+ auto reader_fut = OpenReaderAsync(source_, *format_, options(),
+ internal::GetCpuThreadPool(),
options()->pool);
+ auto reader_gen = GeneratorFromReader(std::move(reader_fut));
+ return MakeSimpleGeneratorIterator(std::move(reader_gen));
+ }
+
+ Future<RecordBatchVector> SafeExecute(internal::Executor* executor) override
{
+ auto reader_fut =
+ OpenReaderAsync(source_, *format_, options(), executor,
options()->pool);
Review comment:
Ahh, I see now - before, while a cpu_executor was passed around in
StreamingReader, it wasn't actually used. (MakeStreamingReader accepts an
executor, but never uses it.) Now it is used, but it'll be a SerialExecutor, so
we remain single-threaded as desired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
