[
https://issues.apache.org/jira/browse/FLINK-41027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41027:
-----------------------------------
Labels: pull-request-available (was: )
> [format/avro] Add an option to map time-millis and time-micros to INT and
> BIGINT
> --------------------------------------------------------------------------------
>
> Key: FLINK-41027
> URL: https://issues.apache.org/jira/browse/FLINK-41027
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Reporter: Pritam Kumar
> Assignee: Pritam Kumar
> Priority: Minor
> Labels: pull-request-available
>
> h2. What
> An option, off by default, to map the Avro logical types {{time-millis}} and
> {{time-micros}} to plain numbers ({{{}INT{}}} and {{{}BIGINT{}}}) instead of
> to {{TIME(3)}} and {{{}TIME(6){}}}.
> h2. Why
> * A Flink {{TIME}} stores milliseconds since midnight in an int, so the
> microseconds of a {{time-micros}} are lost as a {{{}TIME{}}}. As a {{BIGINT}}
> they are kept.
> * A {{TIME(6)}} has no Avro schema: {{AvroSchemaConverter.convertToSchema}}
> refuses a precision above 3. In {{avro-confluent}} with a {{{}schema{}}}, a
> {{time-micros}} field therefore could not be used at all: a {{TIME(6)}}
> column failed before the provided schema was looked at, and a {{BIGINT}}
> column did not match it.
> h2. Change
> * {{AvroSchemaConverter.convertToDataType(String, boolean, boolean)}} and
> {{{}convertToTypeInfo(String, boolean, boolean){}}}: with
> {{{}timeAsNumber{}}}, {{time-millis}} becomes {{INT}} / {{Types.INT}} and
> {{time-micros}} becomes {{BIGINT}} / {{{}Types.LONG{}}}, also nested in
> arrays, maps and nullable unions, with the legacy and the new timestamp
> mapping. The existing overloads keep the current mapping.
> * {{{}AvroToVariantDataConverters.createVariantConverter(Schema,
> boolean){}}}: with {{{}timeAsNumber{}}}, {{time-millis}} stays the Avro int
> instead of being converted to microseconds of day and stored as a long.
> {{time-micros}} is a long either way.
> * {{avro-confluent}} option {{time-mapping.as-number}} (default
> {{{}false{}}}), used when the table schema is checked against the provided
> {{{}schema{}}}. Documented in the English and Chinese pages.
> * Related fix in {{{}RegistryAvroFormatFactory{}}}: the Avro schema of the
> table was built with {{{}Optional.orElse{}}}, which runs even when a
> {{schema}} is provided, so a {{TIME(6)}} column failed there. It now uses
> {{{}orElseGet{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
