[ https://issues.apache.org/jira/browse/ARROW-15280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicola Crane updated ARROW-15280: --------------------------------- Summary: [R] Ignore "*_$folder$" files on S3 in {arrow} R package (was: Ignore "*_$folder$" files on S3 in {arrow} R package) > [R] Ignore "*_$folder$" files on S3 in {arrow} R package > -------------------------------------------------------- > > Key: ARROW-15280 > URL: https://issues.apache.org/jira/browse/ARROW-15280 > Project: Apache Arrow > Issue Type: Improvement > Components: R > Affects Versions: 6.0.1 > Reporter: Bob Rudis > Priority: Minor > > ARROW-4406 notes that: > >Currently reading parquet files generated by Hadoop (EMR) from S3 fails with > >"ValueError: >Found files in an intermediate directory" because of the > >_$folder$ empty files. > This was fixed in the pyarrow but R still has this issue. > The R side does not seem to have something similar to: > {{ def _should_silently_exclude(self, file_name):}} > {{ return (file_name.endswith('.crc') or # Checksums}} > {{ file_name.endswith('_$folder$') or # HDFS directories in S3}} > {{ file_name.startswith('.') or # Hidden files starting with .}} > {{ file_name.startswith('_') or # Hidden files starting with _}} > {{ file_name in EXCLUDED_PARQUET_PATHS)}} -- This message was sent by Atlassian Jira (v8.20.1#820001)