Yin created ARROW-15724: --------------------------- Summary: reduce directory and file IO when reading partition parquet dataset Key: ARROW-15724 URL: https://issues.apache.org/jira/browse/ARROW-15724 Project: Apache Arrow Issue Type: Improvement Reporter: Yin Attachments: pq.py
Hi, It seems Arrow accesses all partitions directories (even the parquet files), including those clearly not matching with the partition key values in the filters. This may cause multiple times difference between accessing one partition directly vs accessing with partition key filters, specially on Network file system, and on local file system when there are lots of partitions, e.g. 1/10th of second vs seconds. Attached Python code to create example dataframe and save parquet datasets with different hive partition structure (/y=/m=/d=, or /y=/m=, or /dk=). And read the datasets with/without filters to reproduce the issue. Observe the run time, and the directories and files which are accessed by the process in Process Monitor on Windows. In the three partition structures, I saw in Process Monitor that all directories are accessed regardless of use_legacy_dataset=True or False. When use_legacy_dataset=False, the parquet files in all directories were opened. The argument validate_schema=False made small time difference, but still opens the partition directories, and it's only supported when use_legacy_dataset=True, and not supported/passed in from pandas read_parquet wrapper API. The /y=/m= is faster since there is no daily partition so less directories and files. There was a related another stackoverflow question and example [https://stackoverflow.com/questions/66339381/pyarrow-read-single-file-from-partitioned-parquet-dataset-is-unexpectedly-slow] and there was a comment on the partition discovery: {quote}It should get discovered automatically. pd.read_parquet calls pyarrow.parquet.read_table and the default partitioning behavior should be to discover hive-style partitions (i.e. the ones you have). The fact that you have to specify this means that discovery is failing. If you could create a reproducible example and submit it to Arrow JIRA it would be helpful. – Pace Feb 24 2021 at 18:55" {quote} Wonder if there was some related Jira here already. I tried passing in partitioning argument, it didn't help. The version of pyarrow used were 1.01, 5, and 7. -- This message was sent by Atlassian Jira (v8.20.1#820001)