xiarixiaoyao commented on PR #5830: URL: https://github.com/apache/hudi/pull/5830#issuecomment-1315020080
> > Either we have some tool for fetching the right avro schema in evolution use cases > > `avroSchema` does not support custom ids. We use them to build mergedInternalSchema to map old type(name) to new type (name) > > > or we keep only the internal schema that is compatible for evolution. > > I tried but it didn't work. There were differences between the origin `avroSchema` and converted `internalSchema` to avro schema. Mb it was a bug with converter. So I just kept the origin `avroSchema` inside `internalSchema`. I don't like this approach and want to revert it. "Leave here the changes only concerning flink" I disagree with putting avro Schema into internal Schema. internalSchema is an independent schema abstraction, should not be bound to avroschema。 **There were differences between the origin `avroSchema` and converted `internalSchema` to avro schema. Mb it was a bug with converter** this is not a bug, The implementation of avroSchemaConvert is different between spark and flink， > > Either we have some tool for fetching the right avro schema in evolution use cases > > `avroSchema` does not support custom ids. We use them to build mergedInternalSchema to map old type(name) to new type (name) > > > or we keep only the internal schema that is compatible for evolution. > > I tried but it didn't work. There were differences between the origin `avroSchema` and converted `internalSchema` to avro schema. Mb it was a bug with converter. So I just kept the origin `avroSchema` inside `internalSchema`. I don't like this approach and want to revert it. "Leave here the changes only concerning flink" not a bug, Both flink and spark have their own schema converters； And the avro schema converted by these two converters are different. This is a gap, we need unity in the future， https://github.com/apache/hudi/pull/6358 do the unify of spark but flink is not included for flink, maybe we can create a new convert function AvroInternalSchemaConverter.buildAvroSchemaFromInternalSchema to convert internalSchema to avro just like flink AvroSchemaConverter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
