kumarpritam863 opened a new pull request, #29522:
URL: https://github.com/apache/flink/pull/29522
## What
An option, off by default, to map the Avro logical types `time-millis` and
`time-micros` to plain numbers (`INT` and `BIGINT`) instead of to `TIME(3)` and
`TIME(6)`.
## Why
- A Flink `TIME` stores milliseconds since midnight in an int, so the
microseconds of a `time-micros` are lost as a `TIME`. As a `BIGINT` they are
kept.
- A `TIME(6)` has no Avro schema: `AvroSchemaConverter.convertToSchema`
refuses a precision above 3. In `avro-confluent` with a `schema`, a
`time-micros` field therefore could not be used at all: a `TIME(6)` column
failed before the provided schema was looked at, and a `BIGINT` column did not
match it.
## Change
- `AvroSchemaConverter.convertToDataType(String, boolean, boolean)` and
`convertToTypeInfo(String, boolean, boolean)`: with `timeAsNumber`,
`time-millis` becomes `INT` / `Types.INT` and `time-micros` becomes `BIGINT` /
`Types.LONG`, also nested in arrays, maps and nullable unions, with the legacy
and the new timestamp mapping. The existing overloads keep the current mapping.
- `AvroToVariantDataConverters.createVariantConverter(Schema, boolean)`:
with `timeAsNumber`, `time-millis` stays the Avro int instead of being
converted to microseconds of day and stored as a long. `time-micros` is a long
either way.
- `avro-confluent` option `time-mapping.as-number` (default `false`), used
when the table schema is checked against the provided `schema`. Documented in
the English and Chinese pages.
- Related fix in `RegistryAvroFormatFactory`: the Avro schema of the table
was built with `Optional.orElse`, which runs even when a `schema` is provided,
so a `TIME(6)` column failed there. It now uses `orElseGet`.
`debezium-avro-confluent` is not changed. Nothing else changes with the
option off, apart from the `orElseGet` fix.
## Testing
- `AvroSchemaConverterTest`: the default mapping, the option, nested and
nullable fields, both timestamp mappings, type info, and that timestamps and
dates are untouched.
- `AvroToVariantDataConvertersTest`: default and option, in a record, a
union and an array.
- `RegistryAvroFormatFactoryTest`: INT/BIGINT columns with the option
(encoder and decoder), the same columns without it (rejected), TIME columns
without it (accepted, which needs the `orElseGet` fix; with `orElse` the test
errors) and with it (rejected).
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Anthropic Claude Sonnet 5.5)
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
