[
https://issues.apache.org/jira/browse/ARROW-13161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17369016#comment-17369016
]
Weston Pace commented on ARROW-13161:
-------------------------------------
David is right, batch readahead is largely ignored as most formats don't
actually do readahead at the file level. I think the exception is parquet and
it does prebuffering so it is difficult to limit it and batch readahead is not
hooked up. It is primarily there as a limit to how much RAM you use. As for
fragment readahead I agree it sounds like what you are describing is a fragment
readahead of 0.
> AFAIK, the ScanBatches method is supposed to be lazy and return an iterator
> and batches will be read out only when Next() is called on the iterator.
No, that is not how it was designed, as you have observed. The scanner is
designed to keep scanning in the background. Can you explain a bit more how
this breaks the iterator mapping with Spark?
What happens if you use the async scanner but set use_threads to false?
I think it would not be a far reach to get "don't read any batches until
ScanBatches is called at least once" but it would be harder to get "don't do
any readahead at all". Also, the scanner is currently being migrated to use an
ExecPlan for filtering/projection and that will create its own pressure.
[~bkietz]
> [C++][Dataset] Allow setting FragmentReadahead to 0 in ScannerBuilder
> ---------------------------------------------------------------------
>
> Key: ARROW-13161
> URL: https://issues.apache.org/jira/browse/ARROW-13161
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Jayjeet Chakraborty
> Priority: Major
> Labels: dataset, datasets
>
> I have an application where I need to set fragment readahead to 0. But, looks
> like for some reason the ScannerBuilder does not allow setting the fragment
> readahead to 0 [1]. It would be very helpful to know why it is that way and
> if a PR lifting that restriction would be accepted because a docstring
> mentions that users can set fragment readahead to 0 if they want [2].
> [1]https://github.com/apache/arrow/blob/master/cpp/src/arrow/dataset/scanner.cc#L864
> [2]https://github.com/apache/arrow/blob/998a2a1668ea57a49d85fbb38f7f0e7eb94c29db/cpp/src/arrow/dataset/scanner.h#L93
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
