gruuya opened a new issue, #9098: URL: https://github.com/apache/arrow-rs/issues/9098
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** With https://github.com/apache/arrow-rs/issues/8004 and https://github.com/apache/arrow-rs/issues/8012 the `parquet`'s `ArrowWriter` became more flexible in terms of what array types it can accept and write. Still this doesn't seem sufficient for a lot of cases. For instance, consider Iceberg, where all fields (including nested ones) should be marked with `PARQUET_FIELD_ID_META_KEY` metadata. When a non-trivial plan involving nested types (e.g. lists) is constructed to stream the batches to the writer, it is not straight-forward to align the list's inner field metadata to what is expected. DataFusion does provide `alias_with_metadata` but this will only align the root level fields, not the nested ones. Consequently an error gets thrown, e.g. ``` ArrowError("Incompatible type. Field 'a' has type List(non-null Int32, metadata: {\"PARQUET:field_id\": \"1\"}), array has type List(non-null Int32)") ``` even though the types are logically equivalent. **Describe the solution you'd like** Relax `LevelInfoBuilder::types_compatible` further, by ignoring nested field names and metadata when comparing data types (i.e. using `DataType::equals_datatype` instead of strict equality). **Describe alternatives you've considered** Map each batch in the stream produced by the DataFusion plan, such that all batch arrays with nested data types are aligned with the target fields. This is unwieldy, and the only thing it does is place `PARQUET_FIELD_ID_META_KEY` in metadata of nested fields, so that the `LevelInfoBuilder::types_compatible` check passes. **Additional context** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
