[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[ https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786078#comment-17786078 ] Mingliang Liu commented on FLINK-30483: --- Just a heads up there is a FLIP for this [FLIP-378|https://cwiki.apache.org//confluence/display/FLINK/FLIP-378%3A+Support+Avro+timestamp+with+local+timezone] and related discussions can happen in the maillist and/or related Jira FLINK-33198 > Make Avro format support for TIMESTAMP_LTZ > -- > > Key: FLINK-30483 > URL: https://issues.apache.org/jira/browse/FLINK-30483 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Mingliang Liu >Assignee: Jagadesh Adireddi >Priority: Major > Labels: pull-request-available, stale-assigned > > Currently Avro format does not support TIMESTAMP_LTZ (short for > TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp > logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP > currently only supports milliseconds, we can make TIMESTAMP_LTZ support > milliseconds first. > A related work is to support microseconds, and there is already > work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the > effort or track that separately for TIMESTAMP_LTZ. > [1] > https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[ https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744423#comment-17744423 ] James Johnston commented on FLINK-30483: I have some high-level questions/concerns about whether the proposed mapping to Avro's local timestamp logical type is really the right one. More details in the PR: https://github.com/apache/flink/pull/21594#issuecomment-1641319553 > Make Avro format support for TIMESTAMP_LTZ > -- > > Key: FLINK-30483 > URL: https://issues.apache.org/jira/browse/FLINK-30483 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Mingliang Liu >Assignee: Jagadesh Adireddi >Priority: Major > Labels: pull-request-available > > Currently Avro format does not support TIMESTAMP_LTZ (short for > TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp > logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP > currently only supports milliseconds, we can make TIMESTAMP_LTZ support > milliseconds first. > A related work is to support microseconds, and there is already > work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the > effort or track that separately for TIMESTAMP_LTZ. > [1] > https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[ https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682980#comment-17682980 ] Dawid Wysakowicz commented on FLINK-30483: -- [~jadireddi] We did a review of the PR in github. Sorry it took so long. Do you mind telling if you're still interested in finalising the PR? Thanks! > Make Avro format support for TIMESTAMP_LTZ > -- > > Key: FLINK-30483 > URL: https://issues.apache.org/jira/browse/FLINK-30483 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Mingliang Liu >Assignee: Jagadesh Adireddi >Priority: Major > Labels: pull-request-available > > Currently Avro format does not support TIMESTAMP_LTZ (short for > TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp > logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP > currently only supports milliseconds, we can make TIMESTAMP_LTZ support > milliseconds first. > A related work is to support microseconds, and there is already > work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the > effort or track that separately for TIMESTAMP_LTZ. > [1] > https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[ https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679450#comment-17679450 ] Jagadesh Adireddi commented on FLINK-30483: --- Hi [~liuml07] , Can you please review PR at your feasible time. > Make Avro format support for TIMESTAMP_LTZ > -- > > Key: FLINK-30483 > URL: https://issues.apache.org/jira/browse/FLINK-30483 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Mingliang Liu >Assignee: Jagadesh Adireddi >Priority: Major > Labels: pull-request-available > > Currently Avro format does not support TIMESTAMP_LTZ (short for > TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp > logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP > currently only supports milliseconds, we can make TIMESTAMP_LTZ support > milliseconds first. > A related work is to support microseconds, and there is already > work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the > effort or track that separately for TIMESTAMP_LTZ. > [1] > https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[ https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654493#comment-17654493 ] Jagadesh Adireddi commented on FLINK-30483: --- Thank you [~liuml07] for letting me know. That should be good. > Make Avro format support for TIMESTAMP_LTZ > -- > > Key: FLINK-30483 > URL: https://issues.apache.org/jira/browse/FLINK-30483 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Mingliang Liu >Priority: Major > Labels: pull-request-available > > Currently Avro format does not support TIMESTAMP_LTZ (short for > TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp > logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP > currently only supports milliseconds, we can make TIMESTAMP_LTZ support > milliseconds first. > A related work is to support microseconds, and there is already > work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the > effort or track that separately for TIMESTAMP_LTZ. > [1] > https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[ https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654214#comment-17654214 ] Mingliang Liu commented on FLINK-30483: --- Thanks [~jadireddi] I'll review from next week. This week I'm oncall. I glimpsed and it looks promising to me overall. [~martijnvisser] I think we need your help for assigning this Jira to Jagadesh and reviewing. I looked again and this does not require FLINK-17224. We can address that separately. > Make Avro format support for TIMESTAMP_LTZ > -- > > Key: FLINK-30483 > URL: https://issues.apache.org/jira/browse/FLINK-30483 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Mingliang Liu >Priority: Major > Labels: pull-request-available > > Currently Avro format does not support TIMESTAMP_LTZ (short for > TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp > logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP > currently only supports milliseconds, we can make TIMESTAMP_LTZ support > milliseconds first. > A related work is to support microseconds, and there is already > work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the > effort or track that separately for TIMESTAMP_LTZ. > [1] > https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[ https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654154#comment-17654154 ] Jagadesh Adireddi commented on FLINK-30483: --- Hi [~liuml07] , Thank you for the response. Filed a PR. Could you please review it. [https://github.com/apache/flink/pull/21594] > Make Avro format support for TIMESTAMP_LTZ > -- > > Key: FLINK-30483 > URL: https://issues.apache.org/jira/browse/FLINK-30483 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Mingliang Liu >Priority: Major > Labels: pull-request-available > > Currently Avro format does not support TIMESTAMP_LTZ (short for > TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp > logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP > currently only supports milliseconds, we can make TIMESTAMP_LTZ support > milliseconds first. > A related work is to support microseconds, and there is already > work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the > effort or track that separately for TIMESTAMP_LTZ. > [1] > https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[ https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17652926#comment-17652926 ] Mingliang Liu commented on FLINK-30483: --- Thanks [~jadireddi] ! I started making changes in a local branch but have not finished. Glad you have a locally working version! I can help review. I do not have permission to assign Jira to you. We need a committer to help us with that. But as we coordinate here, I think you can file the PR now. > Make Avro format support for TIMESTAMP_LTZ > -- > > Key: FLINK-30483 > URL: https://issues.apache.org/jira/browse/FLINK-30483 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Mingliang Liu >Priority: Major > > Currently Avro format does not support TIMESTAMP_LTZ (short for > TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp > logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP > currently only supports milliseconds, we can make TIMESTAMP_LTZ support > milliseconds first. > A related work is to support microseconds, and there is already > work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the > effort or track that separately for TIMESTAMP_LTZ. > [1] > https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[ https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17652795#comment-17652795 ] Jagadesh Adireddi commented on FLINK-30483: --- Hi, I am able to make code changes locally. Can you help me to assign this jira? Will raise a PR. Regards, Jagadesh > Make Avro format support for TIMESTAMP_LTZ > -- > > Key: FLINK-30483 > URL: https://issues.apache.org/jira/browse/FLINK-30483 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Mingliang Liu >Priority: Major > > Currently Avro format does not support TIMESTAMP_LTZ (short for > TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp > logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP > currently only supports milliseconds, we can make TIMESTAMP_LTZ support > milliseconds first. > A related work is to support microseconds, and there is already > work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the > effort or track that separately for TIMESTAMP_LTZ. > [1] > https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[ https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651185#comment-17651185 ] Mingliang Liu commented on FLINK-30483: --- I'm not sure this requires FLINK-17224. Today TIME and TIMESTAMP both support milliseconds (precision 3). It seems to me only TIME supporting precision doesn't work correctly and TIMESTAMP just works fine. TIMESTAMP_LTZ in this JIRA is more related to TIMESTAMP and I guess it's not blocked by FLINK-17224. FLINK-23589 is to add microseconds (precision 6) support for TIMESTAMP. I think it is not blocked by FLINK-17224 either. Meanwhile, FLINK-27331 is to add microseconds precision for TIME which is indeed blocked by FLINK-17224. I have not looked into those work carefully (or test myself) and could have missed something. > Make Avro format support for TIMESTAMP_LTZ > -- > > Key: FLINK-30483 > URL: https://issues.apache.org/jira/browse/FLINK-30483 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Mingliang Liu >Priority: Major > > Currently Avro format does not support TIMESTAMP_LTZ (short for > TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp > logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP > currently only supports milliseconds, we can make TIMESTAMP_LTZ support > milliseconds first. > A related work is to support microseconds, and there is already > work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the > effort or track that separately for TIMESTAMP_LTZ. > [1] > https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[ https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651156#comment-17651156 ] Martijn Visser commented on FLINK-30483: But this ticket also requires FLINK-17224 right? > Make Avro format support for TIMESTAMP_LTZ > -- > > Key: FLINK-30483 > URL: https://issues.apache.org/jira/browse/FLINK-30483 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Mingliang Liu >Priority: Major > > Currently Avro format does not support TIMESTAMP_LTZ (short for > TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp > logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP > currently only supports milliseconds, we can make TIMESTAMP_LTZ support > milliseconds first. > A related work is to support microseconds, and there is already > work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the > effort or track that separately for TIMESTAMP_LTZ. > [1] > https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[ https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651154#comment-17651154 ] Mingliang Liu commented on FLINK-30483: --- [~martijnvisser] I linked FLINK-23589 in the Jira description. I think it's a related issue, but that is for microseconds support (larger precision for TIMESTAMP) type. This Jira tracks TIMESTAMP_LTZ type, which is not supported regardless of precision. > Make Avro format support for TIMESTAMP_LTZ > -- > > Key: FLINK-30483 > URL: https://issues.apache.org/jira/browse/FLINK-30483 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Mingliang Liu >Priority: Major > > Currently Avro format does not support TIMESTAMP_LTZ (short for > TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp > logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP > currently only supports milliseconds, we can make TIMESTAMP_LTZ support > milliseconds first. > A related work is to support microseconds, and there is already > work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the > effort or track that separately for TIMESTAMP_LTZ. > [1] > https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[
https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651147#comment-17651147
]
Mingliang Liu commented on FLINK-30483:
---
Sample code:
{code:java}
public static void main(String[] args) {
final TableEnvironment tableEnv =
TableEnvironment.create(EnvironmentSettings.inStreamingMode());
tableEnv.executeSql(
"CREATE TABLE SinkTable(ts TIMESTAMP_LTZ(3)) "
+ "WITH ('connector' = 'filesystem', 'path' =
'/tmp/flink-avro-test', 'format' = 'avro')");
tableEnv.executeSql("INSERT INTO SinkTable VALUES (PROCTIME())");
tableEnv.executeSql("SELECT * FROM SinkTable").print();
}
{code}
will get following exception:
{code}
Exception in thread "main" java.lang.UnsupportedOperationException: Unsupported
to derive Schema for type: TIMESTAMP_LTZ(3)
at
org.apache.flink.formats.avro.typeutils.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:429)
at
org.apache.flink.formats.avro.typeutils.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:400)
at
org.apache.flink.formats.avro.typeutils.AvroSchemaConverter.convertToSchema(AvroSchemaConverter.java:306)
{code}
> Make Avro format support for TIMESTAMP_LTZ
> --
>
> Key: FLINK-30483
> URL: https://issues.apache.org/jira/browse/FLINK-30483
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.16.0
>Reporter: Mingliang Liu
>Priority: Major
>
> Currently Avro format does not support TIMESTAMP_LTZ (short for
> TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp
> logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP
> currently only supports milliseconds, we can make TIMESTAMP_LTZ support
> milliseconds first.
> A related work is to support microseconds, and there is already
> work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the
> effort or track that separately for TIMESTAMP_LTZ.
> [1]
> https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30483) Make Avro format support for TIMESTAMP_LTZ
[ https://issues.apache.org/jira/browse/FLINK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651146#comment-17651146 ] Martijn Visser commented on FLINK-30483: [~liuml07] Isn't this a duplicate of FLINK-23589? I don't think this ticket can be resolved until FLINK-17224 is fixed. Do you want to pick that up? > Make Avro format support for TIMESTAMP_LTZ > -- > > Key: FLINK-30483 > URL: https://issues.apache.org/jira/browse/FLINK-30483 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Mingliang Liu >Priority: Major > > Currently Avro format does not support TIMESTAMP_LTZ (short for > TIMESTAMP_WITH_LOCAL_TIME_ZONE) type. Avro 1.10+ introduces local timestamp > logic type (both milliseconds and microseconds), see spec [1]. As TIMESTAMP > currently only supports milliseconds, we can make TIMESTAMP_LTZ support > milliseconds first. > A related work is to support microseconds, and there is already > work-in-progress Jira FLINK-23589 for TIMESTAMP type. We can consolidate the > effort or track that separately for TIMESTAMP_LTZ. > [1] > https://avro.apache.org/docs/1.10.2/spec.html#Local+timestamp+%28millisecond+precision%29 -- This message was sent by Atlassian Jira (v8.20.10#820010)
