Vladimir created ARROW-10937: -------------------------------- Summary: ArrowInvalid error on reading partitioned parquet files from S3 (arrow-2.0.0) Key: ARROW-10937 URL: https://issues.apache.org/jira/browse/ARROW-10937 Project: Apache Arrow Issue Type: Bug Components: Python Affects Versions: 2.0.0 Reporter: Vladimir
Hello It looks like pyarrow-2.0.0 has problems in reading parquet could not read partitioned datasets from S3 buckets: {code:java} import s3fs import pyarrow as pa import pyarrow.parquet as pq filesystem = s3fs.S3FileSystem() d = pd.date_range('1990-01-01', freq='D', periods=10000) vals = np.random.randn(len(d), 4) x = pd.DataFrame(vals, index=d, columns=['A', 'B', 'C', 'D']) x['Year'] = x.index.year table = pa.Table.from_pandas(x, preserve_index=True) pq.write_to_dataset(table, root_path='s3://bucket/test_pyarrow.parquet', partition_cols=['Year'], filesystem=filesystem) {code} Now, reading it via pq.read_table: {code:java} pq.read_table('s3://bucket/test_pyarrow.parquet', filesystem=filesystem, use_pandas_metadata=True) {code} Raises exception: {code:java} ArrowInvalid: GetFileInfo() yielded path 'bucket/test_pyarrow.parquet/Year=2017/ffcc136787cf46a18e8cc8f72958453f.parquet', which is outside base dir 's3://bucket/test_pyarrow.parquet' {code} Direct read in pandas: {code:java} pd.read_parquet('s3://bucket/test_pyarrow.parquet'){code} returns empty DataFrame. The issue does not exist in pyarrow-1.0.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)