Hi Mike, The "Infer schema" functionality in NiFi currently generates schemas with the order that will be invalid under Avro 1.9+. I noticed because I've been using that to copy-paste schemas that were "almost right" so I could manually fix them.
I guess that inferred schemas should be fine if the inferring logic is also upgraded by the dependency, but for any cached schemas and my own manually saved schemas this will be somewhat painful. My use case for manually saving inferred schemas is mostly database migration where some inferred "choice" fields are not supported for the target database. Hope this helps, Isha -----Oorspronkelijk bericht----- Van: Mike Thomsen <mikerthom...@gmail.com> Verzonden: donderdag 7 april 2022 12:11 Aan: dev@nifi.apache.org Onderwerp: Need some feedback on how upgrading Avro might cause problems Thread is here for full details: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F5900%23pullrequestreview-922490039&data=04%7C01%7Cisha.lamboo%40virtualsciences.nl%7C8c89ae3e621c4a255c3308da187eea09%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C637849231546931433%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ztcxOWXBkpZFqEh%2Bu%2BG0du5BLUPyZ3WaMxqpeqn%2FexI%3D&reserved=0 It looks like Avro 1.8's schema parser may have been more lenient (or buggy) in enforcing the specification with respect to the ordering of a union for a nullable type. 1.9.X and higher are definitely more opinionated and throw exceptions on schemas that used to work on 1.8.X. For example, this used to be valid: { "name": "message", "type": [ "null", "string" ], "default": "Hello, world" } Now Avro **correctly** throws an exception per the specification. Under 1.8 it did not, and as you can see here, I had to change numerous test schemas in order to make them work under 1.9 to 1.11.0: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnifi%2Fpull%2F5900%2Ffiles%23r835954170&data=04%7C01%7Cisha.lamboo%40virtualsciences.nl%7C8c89ae3e621c4a255c3308da187eea09%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C637849231546931433%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=iBGW6sYZUdxvAADYIB5L2t94RBZH3A5%2BPJMhxuGv8q8%3D&reserved=0 As I said to Matt, I think we're in a "damned if you do, damned if you don't" position here. Thoughts?
