elong0527 opened a new issue, #11634: URL: https://github.com/apache/arrow-rs/issues/11634
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Since the fix for #5152, the Parquet writer always writes the `column_orders` field of `FileMetaData`, and for FLOAT/DOUBLE columns it always writes the `IEEE_754_TOTAL_ORDER` union variant (field 2). There is no `WriterProperties` option to omit the field or to choose a different value: `ThriftMetadataWriter::finish()` computes it unconditionally from the physical type. That makes files with float columns unreadable by older readers whose thrift definitions predate the total-order variant. Concretely, Polars < 1.43.2 fails to open such files at all (`Invalid thrift: bad data` on 1.35.2; `ColumnOrder union has no variant set` on 1.42.1), even though the field is optional in the Parquet format and the data itself is fully compatible. Downstream projects that must support those readers currently have to vendor and patch the parquet crate just to omit one optional footer field. **Describe the solution you'd like** A `WriterProperties` option (file-level, ideally with per-column override) to control the written `ColumnOrder`, e.g.: - write as today (default, no behaviour change), or - omit `column_orders` from `FileMetaData` entirely (readers then report `UNDEFINED`, as for pre-2.4 files). Omitting is semantically safe for writers that emit no statistics/column index, and matches what older writers produced. **Describe alternatives you've considered** - Rewriting the footer after writing: `ParquetMetaDataWriter` funnels into the same `ThriftMetadataWriter::finish()`, which recomputes the field, and the thrift `FileMeta` types are crate-private, so this is not possible through public APIs. - Vendoring/patching parquet (what we are doing downstream in the meantime): works, but is a heavy long-term burden to carry across Arrow/Parquet version bumps for a one-line change. **Additional context** Observed with parquet 60.0.0 (`SerializedFileWriter`, statistics disabled — the field is written regardless). A file written with `column_orders` omitted was verified readable, with identical values, by Polars 1.35.2, current Polars, and PyArrow. Happy to contribute a PR if the API shape above is acceptable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
