[
https://issues.apache.org/jira/browse/FLINK-35439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111144#comment-18111144
]
Bhanu Chander Vallabaneni commented on FLINK-35439:
---------------------------------------------------
I would like to take this one — could a committer assign it to me?
Most of what is needed already exists on both sides, which is why I think this
is a small, contained
change rather than new machinery:
* {{AvroSchemaConverter}} already exposes exactly the two entry points this
asks for as statics:
{{convertToTypeInfo(String avroSchemaString)}} and {{convertToDataType(String
avroSchemaString)}}.
* PyFlink already reaches into that same package over py4j —
{{GenericRecordAvroTypeInfo}} in
{{pyflink/datastream/formats/avro.py}} constructs
{{org.apache.flink.formats.avro.typeutils.GenericRecordAvroTypeInfo}}
directly.
So the work is a thin Python wrapper over those two statics plus converting
what they return into
PyFlink types ({{TypeInformation}} through the existing {{_from_java_type}}
path, and {{DataType}}
for the Table API case the reporter cares about), with tests.
Two things I would want a maintainer's opinion on before writing it, since they
are API choices:
# where it should live — {{pyflink/datastream/formats/avro.py}} next to
{{AvroSchema}}, or somewhere
under {{pyflink/table}} given the Table API is the motivation;
# whether to expose both {{convert_to_type_info}} and {{convert_to_data_type}},
or only the
{{DataType}} one, which is what the {{avro-confluent}} schema-evolution use
case in the
description actually needs.
For context on whether I know this corner: FLINK-40370 was mine, in
{{pyflink/fn_execution/formats/avro.py}}, merged earlier today.
I have also asked about FLINK-34527. If holding two is not welcome, this is the
one I would rather
have — happy to take them one at a time.
> Python Support for AvroSchemaConverter
> --------------------------------------
>
> Key: FLINK-35439
> URL: https://issues.apache.org/jira/browse/FLINK-35439
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
> Affects Versions: 1.17.2, 1.18.1
> Reporter: Greg Wills
> Priority: Major
>
> The java class org.apache.flink.formats.avro.typeutils.AvroSchemaConverter is
> not integrated into the apache-flink Python library (pyflink). My goal is to
> dynamically create a Pyflink table schema from an avro schema, but without a
> Python interface via py4j do so, that task is difficult.
> Especially for the Table API, this would improve schema management and
> evolution practices. Currently, the responsibility is on the user to
> statically create the table schema (used as the Avro reader schema). Schema
> updates can cause this static reader schema that is generated from the table
> schema to be incompatible with the writer schema (when
> consuming/deserializing).
> This is the case specifically for the format avro-confluent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
