cblp opened a new issue, #26855:
URL: https://github.com/apache/datafusion/issues/26855
### Describe the bug
When a Parquet table has field metadata in its schema, such as an Arrow
extension type (`ARROW:extension:name`), the Parquet scan produces the column
without that metadata at execution time. Expressions that read the field
metadata of their arguments see an empty map.
For example, `arrow_metadata(u)` returns `{}` for a Parquet column `u`,
although the table schema has `ARROW:extension:name` on `u`. The same query
over an in-memory table returns the metadata. Extension-aware UDFs that
dispatch on the argument `Field` at execution time are affected the same way.
### To Reproduce
With `datafusion` 55.0.0 from PyPI. `skip_metadata=False` makes schema
inference keep the field metadata; with the default `skip_metadata=True` the
table schema has no field metadata at all, see #26851.
```python
import datafusion
import pyarrow as pa
import pyarrow.parquet as pq
# An extension type unknown to pyarrow, so that pyarrow keeps it as field
metadata
field = pa.field("u", pa.binary(16), metadata={"ARROW:extension:name":
"example.ext"})
table = pa.table({"u": pa.array([b"\x00" * 16], pa.binary(16))},
schema=pa.schema([field]))
pq.write_table(table, "ext.parquet")
ctx = datafusion.SessionContext()
ctx.register_parquet("t", "ext.parquet", skip_metadata=False)
print("table schema:", ctx.table("t").schema().field("u").metadata)
query = "SELECT arrow_metadata(u) AS m FROM t"
print("arrow_metadata(u):", ctx.sql(query).to_pylist())
print(ctx.sql(f"EXPLAIN {query}").to_pylist()[-1]["plan"])
```
Output:
```
table schema: {b'ARROW:extension:name': b'example.ext'}
arrow_metadata(u): [{'m': []}]
DataSourceExec: file_groups={1 group: [[.../ext.parquet]]},
projection=[arrow_metadata(u@0) as m], file_type=parquet
```
Controls on the same version:
| Query | `arrow_metadata(u)` |
|---|---|
| the same table registered with `register_record_batches` |
`[('ARROW:extension:name', 'example.ext')]` |
| `SELECT arrow_metadata(u) FROM (SELECT u FROM t LIMIT 1)` on the Parquet
table, so that the function is not pushed into the scan | `[]` |
So the problem is not the projection pushdown into `DataSourceExec`: the
field produced by the Parquet scan has no metadata.
The extension name is unknown to pyarrow (`example.ext`) only so that
pyarrow keeps it as field metadata. A file written by DataFusion itself with an
`arrow.uuid` column behaves the same way.
### Expected behavior
`arrow_metadata(u)` returns `{ARROW:extension:name: example.ext}`, as for
the in-memory table: the scan produces fields with the metadata of the table
schema.
### Additional context
Related: #26851 (schema inference drops extension types when `skip_metadata
= true`) and #24721 (closed; projection field metadata lost when physical
optimization removes a `ProjectionExec`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
