Joris Van den Bossche created ARROW-17959:
---------------------------------------------
Summary: [C++][Dataset]
Key: ARROW-17959
URL: https://issues.apache.org/jira/browse/ARROW-17959
Project: Apache Arrow
Issue Type: Improvement
Components: C++
Reporter: Joris Van den Bossche
Currently, when reading a subfield of a nested column of a Parquet file using
the Dataset API, we read the full parent column instead of only the requested
field. This should be optimized to only read the field itself.
This was left as a TODO in ARROW-14658
(https://github.com/apache/arrow/pull/11704) which added the initial support
for nested field refs in dataset scanning
(https://github.com/apache/arrow/blob/c29ca51f44eaf41c3a2f6f72e3e23a7b428211c2/cpp/src/arrow/dataset/file_parquet.cc#L240-L246):
{code}
if (field) {
// TODO(ARROW-1888): support fine-grained column projection. We should be
// able to materialize only the child fields requested, and not the entire
// top-level field.
// Right now, if enabled, projection/filtering will fail when they cast the
// physical schema to the dataset schema.
AddColumnIndices(*toplevel, columns_selection);
{code}
Some relevant comments at
https://github.com/apache/arrow/pull/11704#discussion_r749733765. ARROW-1888
was mentioned as a blocker back then, but this is resolved in the meantime.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
