sankalpsthakur opened a new pull request, #1607:
URL: https://github.com/apache/arrow-go/pull/1607

   ### Rationale for this change
   
   Reading a Parquet file with a duration column fails for the whole file in 
v18.8.0:
   
   ```
   no support for reading columns of type: duration
   ```
   
   That covers pandas `timedelta64` columns written through pyarrow, pyarrow 
`duration` columns, and anything else that keeps the Arrow schema in the file. 
Parquet has no duration type, so those writers store INT64 and put the Arrow 
type in the `ARROW:schema` metadata. Before #1095 the column came back as int64.
   
   #1095 made the reader restore the duration type from that metadata 
(`parquet/pqarrow/schema.go:1234`), but the zero-copy case in 
`transferColumnData` (`parquet/pqarrow/column_readers.go:743`) only lists 
`INT32`, `INT64`, `FLOAT32` and `FLOAT64`. A duration column falls through to 
the default case at line 796. The test in #1095 covers the schema step only.
   
   ### What changes are included in this PR?
   
   Add `arrow.DURATION` to that case. `transferZeroCopy` already builds the 
array with the restored type from the INT64 buffer.
   
   ### Are these changes tested?
   
   Yes. `TestReadDurationFromStoredSchema` (`parquet/pqarrow/schema_test.go`, 
next to `TestRestoreDurationFromStoredSchema`) builds a file the way pyarrow 
does, an INT64 column plus `ARROW:schema` holding a duration field, for all 
four units and with a null in the middle. It reads the file with `pqarrow` and 
checks the type and the values. It fails on main with the error above and 
passes with the change.
   
   I did not use a round trip through the Go writer: arrow-go does not write 
durations (`TestUnsupportedTypes` lists `DURATION` as unsupported) and I did 
not want to change that in a bug fix. I did not commit a binary fixture either, 
since there is no Parquet testdata directory under `parquet/pqarrow`.
   
   - `go test ./parquet/pqarrow -run 
'TestReadDurationFromStoredSchema|TestRestoreDurationFromStoredSchema|TestUnsupportedTypes'
 -count=1`: ok
   - `PARQUET_TEST_DATA=$PWD/parquet-testing/data go test ./parquet/pqarrow 
-short -count=1`: ok
   - `go vet ./parquet/pqarrow` and `gofmt -l` on the changed files: clean
   
   ### Are there any user-facing changes?
   
   Yes. Files with duration columns can be read again, as `arrow.DURATION` with 
the unit from the stored schema. Writing durations is still unsupported.
   
   ### AI/LLM disclosure
   
   - AI coding tools were used to help draft or modify code and this PR 
description.
   - I reviewed the complete change, understand the reasoning, and ran the 
reported local tests before submitting.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to