dwsmith1983 opened a new issue, #5936: URL: https://github.com/apache/datafusion-comet/issues/5936
### Describe the bug `remap_physical_schema` in `native/core/src/parquet/schema_adapter.rs` gates the `ParquetMissingFieldIds` rejection on `schema_has_field_ids`, which looks at root fields only and runs only when `spark.sql.parquet.fieldId.read.enabled` is set. Spark's `ParquetReadSupport.getRequestedSchema` uses `containsFieldIds` on the file schema and `ParquetUtils.hasFieldIds` on the requested schema, both recursive, and consults neither flag. Two observable differences follow, both reproducible on `main`: 1. With the default `fieldId.read.enabled=false`, a requested schema carrying `parquet.field.id` metadata read against a file with no ids returns rows in Comet where Spark raises. 2. A file whose ids sit only on nested fields is rejected by Comet where Spark null-fills the missing nested ids through `matchIdField`. ### Steps to reproduce 1. Write a Parquet file with no field ids and read it with a requested schema whose fields carry `parquet.field.id`, native scan on, `fieldId.read.enabled` at its default. Comet returns rows; Spark raises. 2. Write a Parquet file whose ids are only on struct children and read it with field id matching enabled and a requested schema that carries ids at the root. Comet rejects the file; Spark reads it and null-fills the unmatched nested children. ### Expected behavior The gate matches Spark: recursive on both schemas and independent of `fieldId.read.enabled`. ### Additional context #5654 adds the recursive predicate `any_nested_field_has_id` one module away from the root-only `schema_has_field_ids` that makes this decision, and documents the root-only one as deliberate for now. #5786 adds a footer-time validator on the same call site. Neither changes the gate; this issue tracks the gate itself, as asked for in the #5654 review. Nested duplicate resolution is tracked separately in #5884. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
