cblp opened a new issue, #26851:
URL: https://github.com/apache/datafusion/issues/26851
### Describe the bug
When DataFusion infers the schema of Parquet files and
`datafusion.execution.parquet.skip_metadata` is `true` (the default),
`clear_metadata` in `datafusion/datasource-parquet/src/file_format.rs` replaces
the metadata of every top-level field with an empty map. This also removes
`ARROW:extension:name` and `ARROW:extension:metadata`, so a column with an
Arrow extension type (`arrow.uuid`, `arrow.json`, `geoarrow.*`,
`arrow.parquet.variant`, ...) is read as its plain storage type.
As a result, DataFusion cannot read back the extension types it writes: a
table with an extension-typed column written with `COPY ... TO 'file.parquet'`
comes back without the extension type. Extension-aware UDFs and type planners
then see a plain `FixedSizeBinary` / `Binary` / `Struct`.
The problem affects any number of files, not only multi-file tables:
`clear_metadata` runs before `Schema::try_merge` in
`ParquetFormat::infer_schema` whenever `skip_metadata` is set.
This is the item "Parquet schema inference clears all field metadata before
`Schema::try_merge` when `skip_metadata = true`" in the Variant epic #26517,
filed here as a separate issue.
### To Reproduce
With `datafusion` 55.0.0 from PyPI (`clear_metadata` is unchanged on `main`
at e8940acd4):
```python
import datafusion
import pyarrow as pa
import pyarrow.parquet as pq
# An extension type unknown to pyarrow, so that pyarrow keeps it as field
metadata
field = pa.field("u", pa.binary(16), metadata={"ARROW:extension:name":
"example.ext"})
table = pa.table({"u": pa.array([b"\x00" * 16], pa.binary(16))},
schema=pa.schema([field]))
pq.write_table(table, "ext.parquet")
print("written:", pq.read_schema("ext.parquet").field("u").metadata)
ctx = datafusion.SessionContext()
for skip_metadata in [True, False]:
df = ctx.read_parquet("ext.parquet", skip_metadata=skip_metadata)
print(f"skip_metadata = {skip_metadata}:",
df.schema().field("u").metadata)
```
Output:
```
written: {b'ARROW:extension:name': b'example.ext'}
skip_metadata = True: None
skip_metadata = False: {b'ARROW:extension:name': b'example.ext'}
```
The example uses an extension name unknown to pyarrow (`example.ext`):
pyarrow turns a known name such as `arrow.uuid` into its own extension type, so
the metadata would not be visible from Python. The behavior of DataFusion is
the same for any extension name.
### Expected behavior
The extension type keys `ARROW:extension:name` and
`ARROW:extension:metadata` are kept, and the rest of the field metadata is
cleared as before:
```
skip_metadata = True: {b'ARROW:extension:name': b'example.ext'}
```
The extension type is part of the type of a column, not a writer annotation,
so `skip_metadata` should not remove it. Turning `skip_metadata` off is not a
good workaround: it also keeps the schema-level metadata and all other field
metadata, which brings back the merge conflicts that the option exists to avoid.
### Additional context
Proposed fix: in `clear_metadata`, keep the `ARROW:extension:*` keys of each
field and drop the rest. This is the fix suggested in #26517.
Points to agree on:
- **Behavior change.** Files with extension-typed columns will now be read
with the extension type by default. Plans and type coercion over such columns
may change.
- **Merging files.** If a column has an extension type in one file and not
in another, or different `ARROW:extension:metadata` in different files,
`Schema::try_merge` may now fail where it used to succeed. I think an error is
right here because the types differ, but this needs agreement.
- **Nested fields.** `clear_metadata` only touches top-level fields.
Metadata of nested fields (list items, struct children) is not cleared today,
so nested extension types already survive. The fix keeps that as is.
Related: #12644 (extension types), #26810 (extension metadata through
`UNION`).
### Willingness to contribute
I plan to contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
