Sunny-Island commented on issue #39670: URL: https://github.com/apache/arrow/issues/39670#issuecomment-5994777218
I looked into the scan path and think `BasicFragmentEvolution::DevolveSelection` may be a useful place to prevent this inconsistent result. It already identifies fields that are missing from a fragment, and `EvolveBatch` later fills those fields with nulls. Could we reject a missing field there when the Dataset schema declares it non-nullable? This would prevent the scanner from producing a table whose data contradicts its schema, without requiring every fragment to be inspected when the Dataset is created. I realize this would make the scan fail rather than automatically widen the field to nullable, as suggested in the issue. Since the Dataset and projected schemas are already established by scan time, widening may need to happen earlier during discovery. I would be interested in the maintainers' view on which behavior is preferable, particularly for an explicitly supplied schema versus an inferred one. This would address the Dataset scan path; the proposed behavior of `pa.unify_schemas()` may need separate consideration. If the maintainers agree that this behavior should be fixed, I'd be happy to work on a PR. I'd appreciate guidance on whether the preferred behavior is to reject the scan or to widen the inferred schema. @pitrou, since you discussed the related nullability validation issues in #31387 and #41667, would you have a view on the preferred behavior here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
