Emanuel,
What version of NiFi are you using?
I just tested the attached template against the latest, and the FlowFile was
routed to 'invalid' with the explanation:
Records in this FlowFile were invalid for the following reasons: The following
1 fields were missing: [[0]/Records/eventVersion]
Thanks
-Mark
> On Dec 5, 2019, at 7:06 AM, Oliveira, Emanuel <emanuel.olive...@fmr.com>
> wrote:
>
> Hi all,
>
> I been struggling to find a way for ValidateRecord using Avro Schema to force
> mandatory the presence of an array on json payload, problem is if array
> “records” is missing Validate is considering FF valid ☹.
> --objective - Mandatory to have "Records array" with at least "eventVersion"
> - using ValidateRecord > Allow Extra Fields
> - problem im facing is nifi dont trigger payload BAD 1 as invalid!!
>
> How can I make mandatory the Records array ? Is it possible ?
>
> I know I can eventually use a SplitJson JsonPath Expression=$.Records to rid
> off the ARRAY, and also to fial if array "Records" not present.. But I would
> like to have a clean solution using just avro schema, is this possible ?
>
>
>
> --OK - payload GOOD
> {
> "Service": "sssssss",
> "Event": "eeeee",
> "Time": "2019-11-25T16:21:53.280Z",
> "Bucket": "bbb-bbbbb-bbb-bbbbb-bbbbbb",
> "RequestId": "RRRRRRRRRRRRRRRRRR",
> "HostId": "hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh",
> "Records": [{
> "eventVersion": "aaa"
> }
> ]
> }
>
> --NOK - payload BAD 1 - missing "Records" array à BUT
> VALIDATERECORD/AVROSCHEMA SENDS FF TO “valid”!! I want it to be sent
> “invalid” since is not compliant to my avro schema which needs array
> “Records” with element “eventVersion” as 2 mandatory things.
> {
> "Service": "sssssss",
> "Event": "eeeee",
> "Time": "2019-11-25T16:21:53.280Z",
> "Bucket": "bbb-bbbbb-bbb-bbbbb-bbbbbb",
> "RequestId": "RRRRRRRRRRRRRRRRRR",
> "HostId": "hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh",
> "RecordsXXX": [{
> "eventVersion": "aaa"
> }
> ]
> }
>
> --OK - payload BAD 2 - "Records" array present but missing "eventVersion"
> {
> "Service": "sssssss",
> "Event": "eeeee",
> "Time": "2019-11-25T16:21:53.280Z",
> "Bucket": "bbb-bbbbb-bbb-bbbbb-bbbbbb",
> "RequestId": "RRRRRRRRRRRRRRRRRR",
> "HostId": "hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh",
> "Records": [{
> "eventVersionXX": "aaa"
> }
> ]
> }
>
> Its very simple test flow (attachmed the xml template
> ValidateRecord_missing_mandatory_ARRAY_is_VALID_problem.xml) just using
> ValidateRecord with JsonReader/Json Writer:
> <image001.png>
>
>
> Heres ValidateRecord processor + reader/writer controllers:
> Avro schema with just array “Records” and “eventVersion” as min tag on array
> element.
> Using Allow Extra Fields true:
> So im ok having other fields on the root side by side with the array
> “Records”, and also ok to have extra elements inside each array.
> FYI: the real use case im trying to validate AWS SQS message (s3 trigger)
> where I will be interested on several fields, but crafted this simpler
> example just to ask if its possible to force array to be mandatory and with
> at least 1 element ?
> ==========================================================
>
> --ValidateRecord 1.8.0
> Record Reader JsonTreeReader
> Record Writer JsonRecordSetWriter
> Record Writer for Invalid Records
> Schema Access Strategy Use Reader's Schema
> Schema Registry No value set
> Schema Name ${schema.name}
> Schema Text ${avro.schema}
> Allow Extra Fields true
> Strict Type Checking true
>
> --JsonTreeReader 1.8.0 - MANDATORY TO HAVE "Records" ARRAY + "eventVersion"
> on each ARRAY element
> Schema Access Strategy Use 'Schema Text' Property
> Schema Registry
> Schema Name ${schema.name}
> Schema Version
> Schema Branch
> Schema Text
> {
> "name": "MyName",
> "type": "record",
> "namespace": "aa.bb.cc
> <http://aa.bb.cc/>",
> "fields": [{
> "name": "Records",
> "type": {
> "type": "array",
> "items": {
> "name":
> "Records_record",
> "type": "record",
> "fields": [{
> "name":
> "eventVersion",
> "type": "string"
> }
> ]
> }
> }
> }
> ]
> }
> Date Format
> Time Format
> Timestamp Format
>
> --JsonRecordSetWriter 1.8.0
> Schema Write Strategy Do Not Write Schema
> Schema Access Strategy Inherit Record Schema
> Schema Registry
> Schema Name ${schema.name}
> Schema Version
> Schema Branch
> Schema Text { "name": "eventVersion", "type":
> "string" }
> Date Format
> Time Format
> Timestamp Format
> Pretty Print JSON true
> Suppress Null Values Never Suppress
> Output Grouping Array
>
> Thanks in advance,
> Emanuel Oliveira
>
> <ValidateRecord_missing_mandatory_ARRAY_is_VALID_problem.xml>
