Re: Alias Issue
You can easily test two versions of a record schema with the same name if
you use GenericData. You need different names or packages if you're
generating Java code (SpecificData) or using reflection (ReflectData).
One exception is if you want to have both versions in a single file or
schema. In this case, different names within a union are an appropriate
solution.
Thanks,
Doug
On Fri, Oct 12, 2018 at 4:20 PM Jesse Anderson
wrote:
> Doug,
>
> Thanks. Those were the two changes.
>
> A related question to your point about leaving them alone. What is the
> best practice around having two versions around for testing purposes?
> Should you have different namespaces or different record names?
>
> Thanks,
>
> Jesse
>
> On Fri, Oct 12, 2018 at 3:16 PM Doug Cutting wrote:
>
>> Jesse,
>>
>> The record names should match (although Java has been loose about
>> enforcement of that).
>>
>> Also, that should be "aliases", not "alias".
>>
>> What happens if you add:
>> "aliases": ["SimpleCard"]
>> to the second schema, and change the field alias to:
>> "aliases": ["suit"]
>> ?
>>
>> Alternately, you could just leave the name of the record alone, as
>> "SimpleCard" in both cases.
>>
>> Does that help?
>>
>> Thanks,
>>
>> Doug
>>
>>
>>
>>
>> On Fri, Oct 12, 2018 at 11:16 AM Jesse Anderson <
>> je...@bigdatainstitute.io> wrote:
>>
>>> I'm hitting an issue with using an alias. When I do the schema
>>> evolution, I get an exception with "missing required field." As far I know,
>>> this should be a viable schema. Is there something I'm missing?
>>>
>>> Here is the version 1 of the schema:
>>>
>>> {
>>> "namespace": "model",
>>> "type": "record",
>>> "name": "SimpleCard",
>>> "fields": [
>>> {
>>>"name": "suit",
>>>"type": "string",
>>>"doc" : "The suit of the card"
>>> }
>>> ]
>>> }
>>>
>>> Here is version 2 of the schema. The changes are a different record name
>>> and I've aliased the suit/cardsuit field.
>>>
>>> {
>>> "namespace": "model",
>>> "type": "record",
>>> "name": "SimpleCardV2",
>>> "fields": [
>>> {
>>>"name": "cardsuit",
>>>"type": "string",
>>>"doc" : "The suit of the card",
>>>"alias": ["suit"]
>>> }
>>> ]
>>> }
>>>
>>> Here is the stack trace of the error:
>>>
>>> org.apache.avro.AvroTypeException: Found model.SimpleCard, expecting
>>> model.SimpleCardV2, missing required field cardsuit
>>> at
>>> org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:292)
>>> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>>> at
>>> org.apache.avro.io.ResolvingDecoder.readFieldOrder(ResolvingDecoder.java:130)
>>> at
>>> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:215)
>>> at
>>> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
>>> at
>>> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
>>> at
>>> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
>>> at helpers.AvroReaderHelper.getObject(AvroReaderHelper.java:56)
>>>
>>> Thanks,
>>>
>>> Jesse
>>>
>>
Re: Alias Issue
Doug,
Thanks. Those were the two changes.
A related question to your point about leaving them alone. What is the best
practice around having two versions around for testing purposes? Should you
have different namespaces or different record names?
Thanks,
Jesse
On Fri, Oct 12, 2018 at 3:16 PM Doug Cutting wrote:
> Jesse,
>
> The record names should match (although Java has been loose about
> enforcement of that).
>
> Also, that should be "aliases", not "alias".
>
> What happens if you add:
> "aliases": ["SimpleCard"]
> to the second schema, and change the field alias to:
> "aliases": ["suit"]
> ?
>
> Alternately, you could just leave the name of the record alone, as
> "SimpleCard" in both cases.
>
> Does that help?
>
> Thanks,
>
> Doug
>
>
>
>
> On Fri, Oct 12, 2018 at 11:16 AM Jesse Anderson
> wrote:
>
>> I'm hitting an issue with using an alias. When I do the schema evolution,
>> I get an exception with "missing required field." As far I know, this
>> should be a viable schema. Is there something I'm missing?
>>
>> Here is the version 1 of the schema:
>>
>> {
>> "namespace": "model",
>> "type": "record",
>> "name": "SimpleCard",
>> "fields": [
>> {
>>"name": "suit",
>>"type": "string",
>>"doc" : "The suit of the card"
>> }
>> ]
>> }
>>
>> Here is version 2 of the schema. The changes are a different record name
>> and I've aliased the suit/cardsuit field.
>>
>> {
>> "namespace": "model",
>> "type": "record",
>> "name": "SimpleCardV2",
>> "fields": [
>> {
>>"name": "cardsuit",
>>"type": "string",
>>"doc" : "The suit of the card",
>>"alias": ["suit"]
>> }
>> ]
>> }
>>
>> Here is the stack trace of the error:
>>
>> org.apache.avro.AvroTypeException: Found model.SimpleCard, expecting
>> model.SimpleCardV2, missing required field cardsuit
>> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:292)
>> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>> at
>> org.apache.avro.io.ResolvingDecoder.readFieldOrder(ResolvingDecoder.java:130)
>> at
>> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:215)
>> at
>> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
>> at
>> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
>> at
>> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
>> at helpers.AvroReaderHelper.getObject(AvroReaderHelper.java:56)
>>
>> Thanks,
>>
>> Jesse
>>
>
Re: Alias Issue
Jesse,
The record names should match (although Java has been loose about
enforcement of that).
Also, that should be "aliases", not "alias".
What happens if you add:
"aliases": ["SimpleCard"]
to the second schema, and change the field alias to:
"aliases": ["suit"]
?
Alternately, you could just leave the name of the record alone, as
"SimpleCard" in both cases.
Does that help?
Thanks,
Doug
On Fri, Oct 12, 2018 at 11:16 AM Jesse Anderson
wrote:
> I'm hitting an issue with using an alias. When I do the schema evolution,
> I get an exception with "missing required field." As far I know, this
> should be a viable schema. Is there something I'm missing?
>
> Here is the version 1 of the schema:
>
> {
> "namespace": "model",
> "type": "record",
> "name": "SimpleCard",
> "fields": [
> {
>"name": "suit",
>"type": "string",
>"doc" : "The suit of the card"
> }
> ]
> }
>
> Here is version 2 of the schema. The changes are a different record name
> and I've aliased the suit/cardsuit field.
>
> {
> "namespace": "model",
> "type": "record",
> "name": "SimpleCardV2",
> "fields": [
> {
>"name": "cardsuit",
>"type": "string",
>"doc" : "The suit of the card",
>"alias": ["suit"]
> }
> ]
> }
>
> Here is the stack trace of the error:
>
> org.apache.avro.AvroTypeException: Found model.SimpleCard, expecting
> model.SimpleCardV2, missing required field cardsuit
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:292)
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
> at
> org.apache.avro.io.ResolvingDecoder.readFieldOrder(ResolvingDecoder.java:130)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:215)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
> at helpers.AvroReaderHelper.getObject(AvroReaderHelper.java:56)
>
> Thanks,
>
> Jesse
>
