[
https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645523#comment-16645523
]
Brian McKelvey edited comment on AVRO-2199 at 10/10/18 8:57 PM:
----------------------------------------------------------------
This has an unfortunate side effect: it breaks applications from reading
existing data that's been encoded with older versions of schemas that had
incompatible default values specified. The library won't even finish parsing
the old schemas, so you can't decode existing data.
The Avro specification says that when you have a union type, any default value
must be of the first type specified in the union. We have millions of records
of data encoded using a schema where the default value was of the second or
third type in the union instead of the first type. This always worked just fine
before but with this more stringent validation, our application suddenly fails
to decode our existing data because it raises an error while parsing the
writer's schema.
I'm going to try to write a monkey-patch to relax this validation temporarily,
long enough to read all existing data, migrate it to a newer schema, and
re-save it. But that's a lot of time and effort to compensate for a sudden
change in schema validation.
was (Author: theturtle32):
This has an unfortunate side effect: it breaks applications from reading
existing data that's been encoded with older versions of schemas that had
incorrect default values specified. The library won't even finish parsing the
old schemas, so you can't decode existing data.
The Avro specification says that when you have a union type, any default value
must be of the first type specified in the union. We have millions of records
of data encoded using a schema where the default value was of the second or
third type in the union. This always worked before, but with this more
stringent validation, our application suddenly fails to decode our existing
data because it raises an error while parsing the old historical schema.
I'm going to try to write a monkey-patch to relax this validation temporarily,
long enough to read all existing data, migrate it to a newer schema, and
re-save it. But that's a lot of time and effort to compensate for a sudden
change in schema validation.
> Validate that field defaults have the correct type
> ---------------------------------------------------
>
> Key: AVRO-2199
> URL: https://issues.apache.org/jira/browse/AVRO-2199
> Project: Avro
> Issue Type: Improvement
> Components: ruby
> Affects Versions: 1.8.2
> Reporter: Daniel Orner
> Priority: Minor
>
> Currently, on the master branch, when a schema is parsed, it is possible to
> define a field with a type and a default of a totally different type. E.g. if
> the field has type "string", the default can be set to "null".
> I'd like to open a PR which will fix this by running the default through the
> SchemaValidator whenever a new Field is created. See
> [https://github.com/salsify/avro-patches/pull/16]
> cc: [~tjwp]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
