[jira] [Commented] (FLINK-27255) Flink-avro does not support serialization and deserialization of avro schema longer than 65535 characters

[Haizhou Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-27255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17536802#comment-17536802
 ] 

Haizhou Zhao commented on FLINK-27255:
--

Here it is: https://github.com/apache/flink/pull/19719

> Flink-avro does not support serialization and deserialization of avro schema 
> longer than 65535 characters
> -
>
> Key: FLINK-27255
> URL: https://issues.apache.org/jira/browse/FLINK-27255
> Project: Flink
>  Issue Type: Bug
>  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.14.4
>Reporter: Haizhou Zhao
>Assignee: Haizhou Zhao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.16.0, 1.14.5
>
>
> The underlying serialization of avro schema uses string serialization method 
> of ObjectOutputStream.class, however, the default string serialization by 
> ObjectOutputStream.class does not support handling string of more than 66535 
> characters (64kb). As a result, constructing flink operators that 
> input/output Avro Generic Record with huge schema is not possible.
>  
> The purposed fix is two change the serialization and deserialization method 
> of these following classes so that huge string could also be handled.
>  
> [GenericRecordAvroTypeInfo|https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/GenericRecordAvroTypeInfo.java#L107]
> [SerializableAvroSchema|https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/SerializableAvroSchema.java#L55]
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-27255) Flink-avro does not support serialization and deserialization of avro schema longer than 65535 characters

[Haizhou Zhao (Jira)]

Haizhou Zhao created FLINK-27255:


 Summary: Flink-avro does not support serialization and 
deserialization of avro schema longer than 65535 characters
 Key: FLINK-27255
 URL: https://issues.apache.org/jira/browse/FLINK-27255
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.14.4
Reporter: Haizhou Zhao


The underlying serialization of avro schema uses string serialization method of 
ObjectOutputStream.class, however, the default string serialization by 
ObjectOutputStream.class does not support handling string of more than 66535 
characters (64kb). As a result, constructing flink operators that input/output 
Avro Generic Record with huge schema is not possible.

 

The purposed fix is two change the serialization and deserialization method of 
these following classes so that huge string could also be handled.

 

[GenericRecordAvroTypeInfo|https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/GenericRecordAvroTypeInfo.java#L107]

[SerializableAvroSchema|https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/SerializableAvroSchema.java#L55]

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)