[
https://issues.apache.org/jira/browse/PARQUET-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated PARQUET-1627:
--------------------------------------
Fix Version/s: format-2.7.0
> Update specification so that legacy timestamp logical types can be written
> for local semantics as well
> ------------------------------------------------------------------------------------------------------
>
> Key: PARQUET-1627
> URL: https://issues.apache.org/jira/browse/PARQUET-1627
> Project: Parquet
> Issue Type: Task
> Components: parquet-format
> Reporter: Zoltan Ivanfi
> Assignee: Nandor Kollar
> Priority: Major
> Labels: pull-request-available
> Fix For: format-2.7.0
>
>
> The rules for TIMESTAMP forward-compatibility were created based on the
> assumption that TIMESTAMP_MILLIS and TIMESTAMP_MICROS have only been used in
> the instant aka. UTC-normalized semantics so far.
> From this false premise it followed that TIMESTAMPs with local semantics were
> a new type and did not need to be annotated with the old types to maintain
> compatibility. In fact, annotating them with the old types were considered to
> be harmful, since it would have mislead older readers into thinking that they
> can read TIMESTAMPs with local semantics, when in reality they would have
> misinterpreted them as TIMESTAMPs with instant semantics. This would have
> lead to a difference of several hours, corresponding to the time zone offset.
> In reality, however, this misinterpretation of timestamps has already been
> going on for a while, since Arrow annotates local timestamps with
> TIMESTAMP_MILLIS or TIMESTMAP_MICROS.
> To maintain forward compatibilty of local timestamps, the specification
> should allow annotating them with the legacy timestamp logical types.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
