[ https://issues.apache.org/jira/browse/ARROW-17308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17620323#comment-17620323 ]
Alenka Frim commented on ARROW-17308: ------------------------------------- Yes, the new Dataset API doesn't yet support the {{validate_schema}} keyword, but the work is in progress and is tracked with https://issues.apache.org/jira/browse/ARROW-8221 (as it is a duplicated Jira will be closing this issue). The default in {{ParquetDataset}} is still the legacy API and so the keyword {{validate_schema}} should work (without supplying the new [FileSystem|https://arrow.apache.org/docs/dev/python/generated/pyarrow.fs.FileSystem.html#pyarrow.fs.FileSystem], see: [https://github.com/apache/arrow/blob/962121062e4b13c148f24a6d4fa4b1a2f1be0d88/python/pyarrow/parquet/core.py#L1717-L1718], which is relevant for 9.x and I think it was the same in 6.x, could be wrong though) > ValueError: Keyword 'validate_schema' is not yet supported with the new > Dataset API > ----------------------------------------------------------------------------------- > > Key: ARROW-17308 > URL: https://issues.apache.org/jira/browse/ARROW-17308 > Project: Apache Arrow > Issue Type: Bug > Components: Python > Reporter: Abderrahmane Jaidi > Priority: Major > Labels: dataset-parquet-legacy, dataset-parquet-read > > Documentation for PyArrow 6.x and 7.x both indicate that the > `validate_schema` argument is supported in the `ParquetDataset` class. Yet > passing that argument to an instance results in: > ValueError: Keyword 'validate_schema' is not yet supported with the new > Dataset API > Code: > {code:python} > parquet_dataset = pyarrow.parquet.ParquetDataset( > path_or_paths=paths, > validate_schema=validate_schema, > filesystem=filesystem, > partitioning=partitioning, > use_legacy_dataset=False, > ){code} > Docs link: > [https://arrow.apache.org/docs/6.0/python/generated/pyarrow.parquet.ParquetDataset.html] > [https://arrow.apache.org/docs/7.0/python/generated/pyarrow.parquet.ParquetDataset.html] > -- This message was sent by Atlassian Jira (v8.20.10#820010)