[ https://issues.apache.org/jira/browse/ARROW-18114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17623334#comment-17623334 ]
Weston Pace commented on ARROW-18114: ------------------------------------- Yes, I would expect there to be a difference. I'll try and get some time to debug this week and figure out what is going on. > [R] unify_schemas=FALSE does not improve open_dataset() read times > ------------------------------------------------------------------ > > Key: ARROW-18114 > URL: https://issues.apache.org/jira/browse/ARROW-18114 > Project: Apache Arrow > Issue Type: Bug > Components: R > Reporter: Carl Boettiger > Priority: Major > > open_dataset() provides the very helpful optional argument to set > unify_schemas=FALSE, which should allow arrow to inspect a single parquet > file instead of touching potentially thousands or more parquet files to > determine a consistent unified schema. This ought to provide a substantial > performance increase in contexts where the schema is known in advance. > Unfortunately, in my tests it seems to have no impact on performance. > Consider the following reprexes: > default, unify_schemas=TRUE > {code:java} > library(arrow) > ex <- s3_bucket("neon4cast-scores/parquet/terrestrial_30min", > endpoint_override = "data.ecoforecast.org", anonymous=TRUE) > bench::bench_time( > { open_dataset(ex) } > ){code} > about 32 seconds for me. > manual, unify_schemas=FALSE: > {code:java} > bench::bench_time({ > open_dataset(ex, unify_schemas = FALSE) > }){code} > takes about 32 seconds as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)