[ https://issues.apache.org/jira/browse/ARROW-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krisztian Szucs updated ARROW-8221: ----------------------------------- Fix Version/s: 9.0.0 (was: 8.0.0) > [Python][Dataset] Expose schema inference / validation options in the factory > ----------------------------------------------------------------------------- > > Key: ARROW-8221 > URL: https://issues.apache.org/jira/browse/ARROW-8221 > Project: Apache Arrow > Issue Type: Improvement > Components: Python > Reporter: Joris Van den Bossche > Assignee: Joris Van den Bossche > Priority: Major > Labels: dataset, pull-request-available > Fix For: 9.0.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > ARROW-8058 added options related to schema inference / validation for the > Dataset factory. We should expose this in Python in the {{dataset(..)}} > factory function: > - Add ability to pass a user-specified schema with a {{schema}} keyword, > instead of inferring the schema from (one of) the files (to be passed to the > factory finish method) > - Add {{validate_schema}} option to toggle whether the schema is validated > against the actual files or not. > - Expose in some way the number of fragments to be inspected when inferring > or validating the schema. Not sure yet what the best API for this would be. > Some relevant notes from the original PR: > https://github.com/apache/arrow/pull/6687#issuecomment-604394407 -- This message was sent by Atlassian Jira (v8.20.7#820007)