MashrafeeAryan opened a new pull request, #1605:
URL: https://github.com/apache/arrow-go/pull/1605

   ### Rationale for this change
   
   Arrow Go can already save a fixed-size list as a Parquet `LIST`. However, it 
still needs the saved Arrow schema to recover the list size when reading. With 
`FINITE_VECTOR(3)` in the Parquet schema, Arrow Go can read the column as a 
fixed-size list of length 3 even when the file has no saved Arrow schema. We 
are basically trying to save the size in Parquet's own schema.
   
   ### What changes are included in this PR?
   
   - Read and write the `FINITE_VECTOR`
   - Add `pqarrow.WithFiniteVectorLogicalType()` so callers can write eligible 
fixed-size lists with `FINITE_VECTOR`.
   - Reject null elements, `NaN`, and infinity when writing a present vector. A 
whole vector can still be null if the column allows it.
   
   References: [Parquet 
spec](https://parquet.apache.org/docs/file-format/types/logicaltypes/#finite-vectors),
 [Arrow C++ PR](https://github.com/apache/arrow/pull/55348), and [Parquet Java 
PR](https://github.com/apache/parquet-java/pull/3876).
   
   ### Are these changes tested?
   
   Yes. The Parquet schema suite and focused `pqarrow` tests passed on Windows.
   
   ### Are there any user-facing changes?
   
   Yes. Arrow Go can now read FINITE_VECTOR() in the Parquet schema and turn it 
into a fixed-size list of length. To save the FINITE_VECTOR label when writing 
a file, users need to turn on pqarrow.WithFiniteVectorLogicalType(). This 
option is off by default because I believe older Parquet readers may not be 
able to open files with new added FINITE_VECTOR() label.
   
   Codex and Gemini assisted with reviewing it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to