yihua opened a new issue, #864: URL: https://github.com/apache/hudi-rs/issues/864
### Feature Description The write path accepts `Utf8` record keys and maps Arrow types to Avro for the commit metadata schema, but it rejects the string encodings the two most common Python DataFrame libraries produce: - Polars exports string columns as `Utf8View`; `append` fails with `cannot map Arrow type Utf8View to Avro for commit metadata schema`. - pandas with Arrow-backed strings exports `LargeUtf8`; using such a column as the record key fails with `record key field 'id' must be Utf8 for writes`. The same applies to `BinaryView` and `LargeBinary`. The Python binding (#858) casts these four types to `Utf8` / `Binary` before handing batches to core, so Polars and pandas inputs work through Python today. A Rust caller still hits the errors, and the cast costs a copy of every string column on every write. Proposed: accept `Utf8View`, `LargeUtf8`, `BinaryView`, and `LargeBinary` natively in core. The Avro mapping is the same (`string` / `bytes`), the key generator only needs a string view of the column, and Parquet writes all four. Once core accepts them, the binding's cast can go. ### Additional context Found while adding Polars and pandas inputs to the Python write API. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
