[
https://issues.apache.org/jira/browse/ARROW-15879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17506256#comment-17506256
]
Carl Boettiger commented on ARROW-15879:
----------------------------------------
Sorry my minimal example was too minimal. Yes, I had noticed dropping the
partition works, but I cannot then `filter()` on the partition column _before_
collect. Continuing from your reprex, try:
{code:java}
> df %>% filter(gear < 3) %>% collect()
Error in lapply(args, function(x) { : object 'gear' not found {code}
The primary incentive to hive-partition I thought was to benefit from `arrow`'s
ability not to even need to parse those files excluded by the filter. (though
admittedly hive-partition is more of a parquet concept I guess, I was initially
very pleasantly surprised that write_dataset() would even partition in this way
with format="csv", so very cool!)
> [R] passing a schema calls open_dataset to fail on hive-partitioned csv files
> -----------------------------------------------------------------------------
>
> Key: ARROW-15879
> URL: https://issues.apache.org/jira/browse/ARROW-15879
> Project: Apache Arrow
> Issue Type: Improvement
> Components: R
> Affects Versions: 7.0.0, 7.0.1
> Reporter: Carl Boettiger
> Priority: Major
>
> Consider this reprex:
>
> Create a dataset with hive partitions in csv format with write_dataset() (so
> cool!):
>
> {code:java}
> library(arrow)
> library(dplyr)
> path <- fs::dir_create("tmp")
> mtcars %>% group_by(gear) %>% write_dataset(path, format="csv")## works fine,
> even with 'collect()'
> ds <- open_dataset(path, format="csv")## but pass a schema, and things fail
> df <- open_dataset(path, format="csv", schema = ds$schema, skip_rows=1)
> df %>% collect()
> {code}
> In the first call to open_dataset, we don't pass a schema and things work as
> expected.
> However, csv files often need a schema to be read in correctly, particularly
> with partitioned data where it is easy to 'guess' the wrong type. Passing
> the schema though confuses open_dataset, because the grouping column
> (partition column) isn't found on the individual files even though it is
> mentioned in the schema!
> Nor can we just omit the grouping column from the schema, since then it is
> effectively lost from the data.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
