[
https://issues.apache.org/jira/browse/FLINK-30656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807968#comment-17807968
]
Hangxiang Yu edited comment on FLINK-30656 at 1/18/24 3:57 AM:
---------------------------------------------------------------
We should support to remain some messages for TypeSerializerSchemaCompatibility
just like SchemaCompatibility in Avro.
Then every TypeSerializer could defined their own message about compatibility.
I have two proposals:
1. Add new method called TypeSerializerSchemaCompatibility#incompatible and
#compatibleAfterMigration to support message, e.g.
TypeSerializerSchemaCompatibility#incompatible(String message). And deprecated
related old methods.
{code:java}
public static <T> TypeSerializerSchemaCompatibility<T> incompatible(String
message) {
return new TypeSerializerSchemaCompatibility<>(Type.INCOMPATIBLE, message,
null);
} {code}
2. Add a new method called TypeSerializerSchemaCompatibility#withMessage:
{code:java}
private TypeSerializerSchemaCompatibility<T> withMessage(String message) {
this.message = message;
return this;
} {code}
Proposal 1 behaves just like SchemaCompatibility in Avro who forces caller to
add message. But since TypeSerializerSchemaCompatibility is a PublicEvolving
API, maybe we need a FLIP firstly?
Proposal 2 just add a new method so that we will not break change, but every
callers (including some custom-defined TypeSerializers) should call it manually
because it will not fail when compile.
[~leonard] [~Weijie Guo] WDYT?
was (Author: masteryhx):
We should support to remain some messages for TypeSerializerSchemaCompatibility
just like SchemaCompatibility in Avro.
Then every TypeSerializer could defined their own message about compatibility.
I have two proposals:
1. Add new method called TypeSerializerSchemaCompatibility#incompatible and
#compatibleAfterMigration to support message, e.g.
TypeSerializerSchemaCompatibility#incompatible(String message). And deprecated
related old methods.
{code:java}
public static <T> TypeSerializerSchemaCompatibility<T> incompatible(String
message) {
return new TypeSerializerSchemaCompatibility<>(Type.INCOMPATIBLE, message,
null);
} {code}
2. Add a new method called TypeSerializerSchemaCompatibility#withMessage:
{code:java}
private TypeSerializerSchemaCompatibility<T> withMessage(String message) {
this.message = message;
return this;
} {code}
Proposal 1 behaves just like SchemaCompatibility in Avro who forces caller to
add message. But since TypeSerializerSchemaCompatibility is a PublicEvolving
API, maybe we need a FLIP firstly?
Proposal 2 just add a new method so that we will not break change, but every
callers (including some custom-defined TypeSerializers) should call it manually
because it will not fail when compile.
[~leonard] WDYT?
> Provide more logs for schema compatibility check
> ------------------------------------------------
>
> Key: FLINK-30656
> URL: https://issues.apache.org/jira/browse/FLINK-30656
> Project: Flink
> Issue Type: Improvement
> Components: API / Type Serialization System
> Reporter: Hangxiang Yu
> Assignee: Hangxiang Yu
> Priority: Major
>
> Currently, we have very few logs and exception info when checking schema
> compatibility.
> It's difficult to see why the compatibility is not compatible, especially for
> some complicated nested serializers.
> For example, for map serializer, when it's not compatible, we may only see
> below without other information:
> {code:java}
> Caused by: org.apache.flink.util.StateMigrationException: The new state
> serializer
> (org.apache.flink.api.common.typeutils.base.MapSerializer@e95e076a) must not
> be incompatible with the old state serializer
> (org.apache.flink.api.common.typeutils.base.MapSerializer@c33b100f). {code}
> So I think we could add more infos when checking the compatibility.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
