[
https://issues.apache.org/jira/browse/AVRO-419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835465#action_12835465
]
Scott Carey commented on AVRO-419:
----------------------------------
I think that Avro should have good, consistent default behavior, and agree that
this default should probably be lazy. But these defaults also need to be safe
and consistent across all target languages.
However, the details should really be up to the client, fully controlled by
some sort of configuration or annotation. Sometimes a client will want to fail
eagerly long before a specific tuple is encountered that can't be promoted.
Sometimes a client will want an exception to be thrown. Sometimes a client
will want _something else_ to happen -- perhaps a callback or an override for a
default value or something else. Maybe one client thinks its completely fine
to take a double that is larger than MAX_INT and cast it to an int with
truncation to MAX_INT, while another wants an exception in that case, and a
third never wants to down-cast.
There are lots of possibilities, and in the long run I think all those
decisions can be configurable -- both at the schema level and via client
specific overrides.
This can all be achieved with fantastic performance in Java if these 'rules'
were configured up front, and compiled into a parser (fast), static state
machine (faster), or a class generated by asm and compiled 'to the metal' by
the JIT (fastest -- zero overhead for resolving decoders beyond the initial
resolution/compilation cost paid once per schema resolution pair).
> Consistent laziness when resolving partially-compatible changes
> ---------------------------------------------------------------
>
> Key: AVRO-419
> URL: https://issues.apache.org/jira/browse/AVRO-419
> Project: Avro
> Issue Type: Bug
> Components: spec
> Reporter: Raymie Stata
>
> Avro schema resolution is generally "lazy" when it comes to dealing with
> incompatible changes. If the writer writes a union of "int" and "null", and
> the reader expects just an "int", Avro doesn't raise an exception unless the
> writer _actually_ writes a "null" (and the reader attempts to read it).
> This laziness is a powerful feature for supporting "forward compatibility"
> (old readers reading data written by new writers). In the example just
> given, for example, we might decide at some point that a column needs to be
> "nullable" but there's a lot of old code that assumes that it's not. When
> using old code, we can often arrange to avoid sending the old code any new
> records that have null-values in that column. It's powerful to allow new
> writers to write against the nullable schema and allow readers to read those
> records. (For this to be safe, it's also important that this be _checked,_
> i.e., that a run time error is thrown is a bad value is passed to the reader.)
> Avro is lazy in many places (e.g., in the union example just given, and for
> enumerations). But it's not _consistently_ lazy. I propose we comb through
> the spec and make it lazy in all places we can, unless there's a compelling
> reason not to.
> Numeric types is one area where Avro is not consistently lazy. I propose
> that we fairly liberally allow any change from one numeric type to another,
> and raise errors at runtime if bad values are found. An "int" can be changed
> to a "long", for example, and an error is raised when a reader gets an
> out-of-bounds value. A "double" can be changed to an "int", and an error is
> raised if the reader gets a non-integer value or an out-of-bounds value.
> (I'm not sure if there are types beyond numerics where we could be more
> consistently lazy, but I decided to write this issue generically just in
> case.)
> (One might object that these checks are expensive, but note that they are
> only needed when the reader and writer specs don't agree. Thus, if these
> checks are induced, then the system designer _wanted_ these checks, we're
> only adding value here, not inducing costs.)
> I'm not sure if there are other a
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
