[
https://issues.apache.org/jira/browse/ARROW-9676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17174983#comment-17174983
]
Neal Richardson edited comment on ARROW-9676 at 8/10/20, 6:17 PM:
------------------------------------------------------------------
Thanks. Could you possibly provide some minimal code that reproduces this,
possibly with a minimal data file or perhaps generating such a file in your
code? It's hard to debug something if I can't trigger the error locally. You
could start by trying to read a single file rather than a whole dataset and see
if that errors; then maybe you can read it with {{as_data_frame=FALSE}}, slice
out a single row from the Arrow Table, and then write that back out. Or try
reading with pyarrow to see if you can slice down to a smaller sample.
Another option: create an R data.frame containing a data.frame column (itself
maybe containing a data.frame column?), send that to Arrow
({{Table$create(...)}}), and see if calling {{as.data.frame()}} on that crashes
the same way.
Also, if you're trying to read Parquet files with nested data structures, I
don't think that's currently supported, though I don't know how it fails
(hopefully without crashing). See ARROW-1644.
was (Author: npr):
Thanks. Could you possibly provide some minimal code that reproduces this,
possibly with a minimal data file or perhaps generating such a file in your
code? It's hard to debug something if I can't trigger the error locally.
Also, if you're trying to read Parquet files with nested data structures, I
don't think that's currently supported, though I don't know how it fails
(hopefully without crashing). See ARROW-1644.
> [R] Option to import structs as lists
> -------------------------------------
>
> Key: ARROW-9676
> URL: https://issues.apache.org/jira/browse/ARROW-9676
> Project: Apache Arrow
> Issue Type: New Feature
> Components: R
> Affects Versions: 1.0.0
> Environment: Amazon Linux, 32gb of ram
> Reporter: Nick DiQuattro
> Priority: Major
>
> When trying to collect data from a dataset based on parquet files with nested
> structs (column is a struct with 2 structs nested) of moderate size (1Mish
> rows), R crashes. If I add a filter to reduce the number of rows, the data is
> parsed. If I select out the struct column, it works great (up to 21M rows).
> My hunch is the structs resulting in data.frame columns may be the issue. I
> am curious if there's a way to have arrow import structs as lists instead of
> data.frames. Thanks for the direction to here [~neilr8133]!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
