RE: ConvertAvroSchema with unmapped fields

[David Gallagher]

Hi Andy – this is the schema we’re getting from the database query:

  "type":"record",
  "name":"view_Logons",
  "namespace":"any.data",
  "fields":[
    {"name":"id","type":["null","int"]},
    {"name":"timestamp","type":["null","string"]},
    {"name":"workstation","type":["null","string"]},
    {"name":"username","type":["null","string"]},
    {"name":"transaction_type","type":["null","string"]},
    {"name":"result","type":["null","string"]},
    {"name":"message","type":["null","string"]}
    ]
  }


And this is the schema we’re converting to:

{"namespace": "com.cleverdevices.cleverreports.avro",
"type": "record",
"name": "AuthorizationTransaction",
"fields": [
     {"name": "product", "type": ["string", "null"]},
     {"name": "sub_product", "type": ["string", "null"]},
     {"name": "id", "type": "long"},
     {"name": "timestamp", "type": "string"},
     {"name": "workstation", "type": ["string", "null"]},
     {"name": "username", "type": ["string", "null"]},
     {"name": "transaction_type", "type": ["string"]},
     {"name": "result", "type": "string"},
     {"name": "message", "type": ["string", "null"]}
]
}

Thanks,

Dave

From: Andy LoPresto <alopre...@apache.org>
Sent: Thursday, August 23, 2018 1:28 PM
To: users@nifi.apache.org
Subject: Re: ConvertAvroSchema with unmapped fields

Hi Dave,

Can you provide the two schemas (redact anything necessary). There is a way to 
specify an “optional” field [1] by setting the type to an array of null and the 
type you support. You can also specify a default value if you wish, which will 
be set for records that do not contain a value there:


{

  "type": "record",

  "name": "Address",

  "fields" : [

    {"name": "streetNumber", "type": "int"},

    {"name": "aptNumber", "type": ["null", "int"]}, // optional apt number

    {"name": "country", "type": "string", "default": "US"}, // default country

  ]

}


[1] https://avro.apache.org/docs/1.8.1/spec.html#schema_record


Andy LoPresto
alopre...@apache.org<mailto:alopre...@apache.org>
alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Aug 23, 2018, at 10:07 AM, David Gallagher 
<dgallag...@cleverdevices.com<mailto:dgallag...@cleverdevices.com>> wrote:

Hi – I’ve got a scenario where I’m trying to convert the implicit AVRO schema 
associated with a database record to a different AVRO schema. I’m trying to use 
the ConvertAvroSchema processor, but it won’t validate because there is an 
‘unmapped’ field that exists in the second schema but not the first. Now, I 
could work around this issue by including a constant in the database record, 
but I would prefer to have a null or default value supplied by the second 
schema instead. Is there some setting I’m missing that would let me specify the 
behavior for unmapped fields?

Thanks,

Dave