[
https://issues.apache.org/jira/browse/ARROW-17459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599053#comment-17599053
]
Micah Kornfield commented on ARROW-17459:
-----------------------------------------
[~arthurpassos] awesome, nice work. IMO, I don't think we can change the
default to LargeBinary/LargeString, as you can see based on the assertion there
is an expectation that types produced match the schema. Also for most
use-cases they aren't necessary, and require extra memory (and might be less
well supported in other implementations).
I think the right way of approach this is to have an option users can set
(maybe one for each type) that will work on two levels:
1. Translate any non-large types in the schema to their large variants.
2. Make the changes at the decoder level that you have already done.
So we keep the assertion but if users run into this issue we can provide
guidance on how to set this.
> [C++] Support nested data conversions for chunked array
> -------------------------------------------------------
>
> Key: ARROW-17459
> URL: https://issues.apache.org/jira/browse/ARROW-17459
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Arthur Passos
> Assignee: Arthur Passos
> Priority: Blocker
>
> `FileReaderImpl::ReadRowGroup` fails with "Nested data conversions not
> implemented for chunked array outputs". It fails on
> [ChunksToSingle]([https://github.com/apache/arrow/blob/7f6b074b84b1ca519b7c5fc7da318e8d47d44278/cpp/src/parquet/arrow/reader.cc#L95])
> Data schema is:
> {code:java}
> optional group fields_map (MAP) = 217 {
> repeated group key_value {
> required binary key (STRING) = 218;
> optional binary value (STRING) = 219;
> }
> }
> fields_map.key_value.value-> Size In Bytes: 13243589 Size In Ratio: 0.20541047
> fields_map.key_value.key-> Size In Bytes: 3008860 Size In Ratio: 0.046667963
> {code}
> Is there a way to work around this issue in the cpp lib?
> In any case, I am willing to implement this, but I need some guidance. I am
> very new to parquet (as in started reading about it yesterday).
>
> Probably related to: https://issues.apache.org/jira/browse/ARROW-10958
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
