[ https://issues.apache.org/jira/browse/ARROW-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17157240#comment-17157240 ]
Joris Van den Bossche commented on ARROW-9459: ---------------------------------------------- One question is, if we do this, where / of which object it should be an option. Since we are collecting statistics in multiple places (during discovery in ParquetDatasetFactory, but also in SplitByRowGroups / EnsureMetadata of the materialized Fragments, as when actually reading as well, I think), it could be a reader option of the format? (ParquetFileFormat) > [C++][Dataset] Make collecting/parsing statistics optional for ParquetFragment > ------------------------------------------------------------------------------ > > Key: ARROW-9459 > URL: https://issues.apache.org/jira/browse/ARROW-9459 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ > Reporter: Joris Van den Bossche > Priority: Major > Labels: dataset, dataset-dask-integration > > See some timing checks here: > https://github.com/dask/dask/pull/6346#issuecomment-656548675 > Parsing all statistics, even from a centralized {{_metadata}} file, can be > quite expensive. If you know in advance that you are not going to use them > (eg you are only going to do filtering on the partition fields, and otherwise > read all data), it could be nice to have an option to disable parsing > statistics. > cc [~rjzamora] [~bkietz] [~fsaintjacques] -- This message was sent by Atlassian Jira (v8.3.4#803005)