[ https://issues.apache.org/jira/browse/ARROW-8245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072793#comment-17072793 ]
Joris Van den Bossche commented on ARROW-8245: ---------------------------------------------- You can already start with a PR fixing it in the python ParquetDataset code and adding a test. Then depending on whether that PR or ARROW-8039 gets merged first, we can parametrize the test to test it for both the python and new datasets implementation. > [Python][Parquet] Skip hidden directories when reading partitioned parquet > files > -------------------------------------------------------------------------------- > > Key: ARROW-8245 > URL: https://issues.apache.org/jira/browse/ARROW-8245 > Project: Apache Arrow > Issue Type: Bug > Components: Python > Reporter: Caleb Overman > Priority: Minor > Labels: parquet > Fix For: 0.17.0 > > > When writing a partitioned parquet file Spark can create a temporary hidden > {{.spark-staging}} directory within the parquet file. Because it is a > directory and not a file, it is not skipped when trying to read the parquet > file. Pyarrow currently only skips directories prefixed with {{_}}. -- This message was sent by Atlassian Jira (v8.3.4#803005)