[
https://issues.apache.org/jira/browse/ARROW-15492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17490695#comment-17490695
]
Micah Kornfield commented on ARROW-15492:
-----------------------------------------
OK so it isn't a bug with missing the logical type.
> So I'm not sure whether Arrow should adjust int96 type data stored in parquet
> file with the local time zone?
The only solution seems to be to pass in an optional target timezone to convert
to. There is no guarantee local time aligns with the writer's timezone. I
think the C++ library has started vendoring the necessary utilities to do the
time zone conversions.
An alternative could also be to provide additional metadata that consumers
could use to determine the source and pad as necessary outside of pyarrow.
> [Python] handle timestamp type in parquet file for compatibility with older
> HiveQL
> ----------------------------------------------------------------------------------
>
> Key: ARROW-15492
> URL: https://issues.apache.org/jira/browse/ARROW-15492
> Project: Apache Arrow
> Issue Type: New Feature
> Affects Versions: 6.0.1
> Reporter: nero
> Priority: Major
>
> Hi there,
> I face an issue when I write a parquet file by PyArrow.
> In the older version of Hive, it can only recognize the timestamp type stored
> in INT96, so I use table.write_to_data with `use_deprecated
> timestamp_int96_timestamps=True` option to save the parquet file. But the
> HiveQL will skip conversion when the metadata of parquet file is not
> created_by "parquet-mr".
> [hive/ParquetRecordReaderBase.java at
> f1ff99636a5546231336208a300a114bcf8c5944 · apache/hive
> (github.com)|https://github.com/apache/hive/blob/f1ff99636a5546231336208a300a114bcf8c5944/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java#L137-L139]
>
> So I have to save the timestamp columns with timezone info(pad to UTC+8).
> But when pyarrow.parquet read from a dir which contains parquets created by
> both PyArrow and parquet-mr, Arrow.Table will ignore the timezone info for
> parquet-mr files.
>
> Maybe PyArrow can expose the created_by option in pyarrow({*}prefer{*},
> parquet::WriterProperties::created_by is available in the C++ ).
> Or handle the timestamp type with timezone which files created by parquet-mr?
>
> Maybe related to https://issues.apache.org/jira/browse/ARROW-14422
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
