Thanks, Mark! Appreciate you retaking another look at this! From: Mark Payne [mailto:marka...@hotmail.com] Sent: 26 January 2022 00:21 To: users <users@nifi.apache.org> Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON
Thanks Nathan, the template is helpful. I put up a PR that I believe addresses the issues. Thanks -Mark On Jan 25, 2022, at 7:21 AM, nathan.engl...@bt.com<mailto:nathan.engl...@bt.com> wrote: Hey Mark, Apologies if this is more of a dev mailing list issue. Happy to go across to it if needs be. I've been testing the fix you've implemented for this and noticed a slight issue when the default value should be null. It took a little while to work out what was going on, but I think I've cracked it. I have attached a flow I've been using to test this. So in AvroTypeUtil on line 630, it's creating the field as it doesn't exist. Instead of setting the default value to null, it is set to org.apache.avro.JsonProperties$Null, which throws an error when it tries to encode as Avro. For example, the Null String & Byte Type generator in the attached flow reproduces this. In addition to this, I did a quick bit of testing with int and long types with default values in the Avro Schema set to 0. So, it looks like Long fields in the schema have the default values set as Integers? I have reproduced it with the attached flow's Int & Long Type generator. Of course, this isn't an exhaustive list of the possible Avro types, but it does seem the changes made to the Avro Util class since v1.12.1 may be causing issues? Or at least it feels that way to me? The only way I see forward is to restore the logic that was previously in place? I have also created a bug NIFI-9629 [1]. Kind Regards, Nathan https://issues.apache.org/jira/browse/NIFI-9629<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-9629&data=04%7C01%7Cnathan.english%40bt.com%7C963069953aac42c2b32f08d9e0489c01%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637787424732462221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=y9Mg2%2BktlI99iCrDWPA%2Bwwp5YDnKu0l9pjGRqOs9gB8%3D&reserved=0> From: English,N,Nathan,VIR R Sent: 20 January 2022 19:53 To: users@nifi.apache.org<mailto:users@nifi.apache.org> Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON Hey Mark, You beat me to the fix, Thanks for taking your time to look at it! I appreciate that one. Cheers! Nathan ________________________________ From: Mark Payne <marka...@hotmail.com<mailto:marka...@hotmail.com>> Sent: Wednesday, January 19, 2022 6:54:23 PM To: users@nifi.apache.org<mailto:users@nifi.apache.org> <users@nifi.apache.org<mailto:users@nifi.apache.org>> Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON Thanks Nathan. I created a Jira [1] for this. I was able to easily replicate with your template. Thanks for including that. Just put up a Pull Request for it, as well. Thanks -Mark [1] https://issues.apache.org/jira/browse/NIFI-9594<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-9594&data=04%7C01%7Cnathan.english%40bt.com%7C963069953aac42c2b32f08d9e0489c01%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637787424732462221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=4pqmI4cJrnAXP3yxv0cxIBoXm%2F50CLC7aAsfVKx5Ij0%3D&reserved=0> On Jan 18, 2022, at 4:49 AM, nathan.engl...@bt.com<mailto:nathan.engl...@bt.com> wrote: Hi There, We've recently upgraded to NiFi V1.15.2 from V1.12.1 and have noticed a difference between the JSON Tree Reader Controller Service, which is a slight annoyance. When we transform our records, we have some fields that will exist or won't after the transformation, depending on whether there is a value for the field. Previously with NiFi v1.12.1, when we would convert these records to Avro using the Avro Writer Controller and JSON Tree Reader Services (Both with Schemas Set), these fields would be created and set with the default value in the Avro Schema. But in NiFi v1.15.2, the conversion fails with "null of boolean in field field_name of Avro.Schema.Name", it doesn't matter which field, type or default value. It all seems to fail if the field doesn't exist. Interestingly, there isn't an issue when I set the JSON Tree Reader to Infer Schema instead of 'use Schema Name' with a Schema Registry. Annoyingly before we convert to Avro from JSON, we go through a Validate Schema Processor, which uses the same Schema, and JSON Tree Reader but instead uses a JSON Writer and successfully validates the record against Schema! It seems similar to the NIFI-9335 issue on Jira [1], but maybe with the JSON Tree Reader Controller service instead? I have attached a sample flow to reproduce the issue. I didn't know if it was worth me raising a bug ticket? Kind Regards, Nathan [1] https://issues.apache.org/jira/browse/NIFI-9335<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-9335&data=04%7C01%7Cnathan.english%40bt.com%7C963069953aac42c2b32f08d9e0489c01%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637787424732618451%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=07S5qtJ7CVH0ZHbCuqsvBRxZWISGy%2BbYlgLqDs%2FWPRc%3D&reserved=0> <NiFi_Field_Conversion_Issue.xml> <Avro_Default_Value_Conversion_Issue_-_20220125.xml>
