stackedsax commented on issue #32766:
URL: https://github.com/apache/arrow/issues/32766#issuecomment-5981334511
I don't think this is a bug in the writer. `time64[ns]` has been written
with the Parquet `TIME(NANOS)` logical type since ARROW-6077 (pyarrow 0.15,
2019), and this is covered in `arrow_schema_test.cc` and
`arrow_reader_writer_test.cc`. On pyarrow 24.0.0:
```python
import io, pyarrow as pa, pyarrow.parquet as pq
t = pa.table({"TIME": pa.array([1, 2], type=pa.time64("ns"))})
buf = io.BytesIO()
pq.write_table(t, buf)
c = pq.ParquetFile(buf).metadata.schema.column(0)
print(c.physical_type, c.converted_type, c.logical_type)
# INT64 NONE Time(isAdjustedToUTC=false, timeUnit=nanoseconds)
```
What the report is showing is an Avro-style schema dump (parquet-mr /
parquet-tools), and that converter works from the legacy *ConvertedType*, not
the newer *LogicalType*. Parquet only has `TIME_MILLIS` and `TIME_MICROS`
converted types; there is no `TIME_NANOS`, so a `TIME(NANOS)` column has no
ConvertedType to fall back to and the tool prints a bare `long`. The other
columns in the same dump confirm this: `DATE` shows `logicalType: date` and
`TIMESTAMP` shows `timestamp-micros` because those do have ConvertedType
equivalents.
If you need legacy readers to see the annotation, write the column as
`time64[us]` (which maps to `TIME_MICROS`) rather than nanoseconds. Note that
since 22.0.0 (#47316) `TIME_MICROS` is only emitted when
`write_time_adjusted_to_utc=True`, because the legacy converted types imply UTC
semantics.
@thisisnic, @rok I think this can be closed as not a bug (behaviour of
parquet-mr's Avro schema converter, not the Arrow writer).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
