MashrafeeAryan opened a new pull request, #1605: URL: https://github.com/apache/arrow-go/pull/1605
### Rationale for this change Arrow Go can already save a fixed-size list as a Parquet `LIST`. However, it still needs the saved Arrow schema to recover the list size when reading. With `FINITE_VECTOR(3)` in the Parquet schema, Arrow Go can read the column as a fixed-size list of length 3 even when the file has no saved Arrow schema. We are basically trying to save the size in Parquet's own schema. ### What changes are included in this PR? - Read and write the `FINITE_VECTOR` - Add `pqarrow.WithFiniteVectorLogicalType()` so callers can write eligible fixed-size lists with `FINITE_VECTOR`. - Reject null elements, `NaN`, and infinity when writing a present vector. A whole vector can still be null if the column allows it. References: [Parquet spec](https://parquet.apache.org/docs/file-format/types/logicaltypes/#finite-vectors), [Arrow C++ PR](https://github.com/apache/arrow/pull/55348), and [Parquet Java PR](https://github.com/apache/parquet-java/pull/3876). ### Are these changes tested? Yes. The Parquet schema suite and focused `pqarrow` tests passed on Windows. ### Are there any user-facing changes? Yes. Arrow Go can now read FINITE_VECTOR() in the Parquet schema and turn it into a fixed-size list of length. To save the FINITE_VECTOR label when writing a file, users need to turn on pqarrow.WithFiniteVectorLogicalType(). This option is off by default because I believe older Parquet readers may not be able to open files with new added FINITE_VECTOR() label. Codex and Gemini assisted with reviewing it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
