sovsparrow opened a new pull request, #24387:
URL: https://github.com/apache/datafusion/pull/24387
## Which issue does this PR close?
- Closes #24381.
- Closes #12852.
## Rationale for this change
When a Parquet file has multiple columns with the same name and compatible
types, DataFusion merges them during schema inference. The scan succeeds
without warning but returns only one of those columns.
PyArrow can produce such a file via its public API:
```python
left = pa.table({"id": [1, 2, 3], "value": [10, 20, 30]})
right = pa.table({"id": [1, 2, 3], "value": [100, 200, 300]})
joined = left.join(right, keys="id") # left_suffix/right_suffix default to
None
pq.write_table(joined, path) # writes id, value, value
```
DataFusion 54.0.0 returns ['id', 'value'] for this file; the [100, 200, 300]
column is missing.
Other readers either preserve the data or reject the file:
| Reader | Result |
| --- | --- |
| `pyarrow` `ParquetFile.read()` | keeps all three columns |
| `pyarrow` `pq.read_table()` | `ArrowInvalid: Multiple matches for
FieldRef.Name(value)` |
| `pyarrow` dataset | `ArrowInvalid: Can't unify schema with duplicate field
names` |
| `duckdb` | keeps all three, renames the second to `value_1` |
| `polars` | `DuplicateError` |
| `datafusion` | **succeeds, missing column(s)** |
The same bug is behind two issues #24381 and #12852. CSV and Parquet schema
inference both pass each file's schema to `Schema::try_merge`, which matches
fields by name. Duplicate names in one file are merged before the scan. Thus,
column(s) disappear silently.
## What changes are included in this PR?
This PR adds `ensure_unique_field_names` to `datafusion-datasource`. It is
called for each inferred CSV and Parquet schema before the merge. It uses the
existing `SchemaError::DuplicateUnqualifiedField` error and adds the file
location to the message.
Duplicate names are rejected. The PR does not rename columns or change how
fields with unique names are merged across files.
## Are these changes tested?
Yes. Two regression tests in order to cover the CSV and Parquet paths:
- `datafusion/core/src/datasource/file_format/parquet.rs` —
`infer_schema_rejects_duplicate_field_names`
- `datafusion/core/src/datasource/file_format/csv.rs` —
`infer_schema_rejects_duplicate_header_names`
Both tests create their inputs in a temporary directory; no fixtures are
added. Both fail on `main` and pass with this change. On `main`, the CSV test
infers `Schema { fields: [id, value] }` from a three-column header, matching
the behavior reported in #12852.
## Are there any user-facing changes?
Yes. CSV and Parquet schema inference now returns a clear error when an
inferred
file schema repeats a field name. The error names the file and the repeated
column instead of returning an incomplete schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
