If you know the input and output schemas, you should be able to use UpdateRecord for this. You would have a user-defined property for each output field (including flattened field names), whose value would be the RecordPath to the field in the map. I believe for any fields that are in the input schema but not the output schema, they are ignored (but don't quote me on this, give it a try ;) so you could effectively "convert" the record by updating outgoing fields with the values from the nested fields in the input record.
Regards, Matt On Wed, Oct 11, 2017 at 12:10 PM, Bryan Bende <[email protected]> wrote: > Casting string to integer wouldn't be a problem, but flattening a map would > be. > > I think there was a previous thread where a FlattenRecord processor > was discussed, and I believe there was agreement that it would be > useful, but it just hasn't been developed yet. > > If you had FlattenRecord then you would probably bring the data in > from ConsumeKafkaRecord keeping the original schema (using a writer > with "inherit schema from reader") and then later in the flow do a > FlattenRecord. > > As a current option, you might be able to do the flattening yourself > with an ExecuteScript processor, or maybe a scripted record writer > that did the flattening on the way out. > > > On Wed, Oct 11, 2017 at 11:22 AM, Chris Herssens > <[email protected]> wrote: >> Hello Bryan, >> >> In my case the schemas are not compatible, In Schema A I have a map type and >> I want to flatten this map. Also I want to cast string to integer. >> >> Chris >> >> On Wed, Oct 11, 2017 at 5:03 PM, Bryan Bende <[email protected]> wrote: >>> >>> Chris, >>> >>> Any processor that uses a record reader and record writer can >>> inherently do schema conversion by using schema A for the reader, and >>> schema B for the writer, assuming the schemas are compatible. >>> >>> Compatible in this sense would mean one of the following... >>> >>> - Schema B has the same field names as schema A, but with some of the >>> filed types being different >>> - Schema B has a subset of the fields in schema A, possibly some >>> changing type as above >>> - Schema B has additional fields and they have default values since >>> they won't exist in the records coming from schema A >>> >>> If you have Avro data in Kafka that already has the schema embedded in >>> it, then you can use ConsumeKafkaRecord with an AvroReader and set the >>> Schema Access Strategy to "Embedded Avro", and then use a >>> AvroRecordSetWriter and set the Schema Access Strategy to one of the >>> other options like Schema Name (which needs a schema registry) or >>> Schema Text which allows you to enter in a full schema. >>> >>> You could also do the same thing anywhere else in your flow using >>> ConvertRecord. >>> >>> Thanks, >>> >>> Bryan >>> >>> >>> On Wed, Oct 11, 2017 at 7:55 AM, Chris Herssens >>> <[email protected]> wrote: >>> > Hello All, >>> > >>> > I would like to convert an avro schema to another avro schema. Since >>> > Nifi >>> > reads the avro data from kafka, I can't use the ConvertAvroSchema >>> > processor. >>> > Which processor can I use ? >>> > Is it possible to use ConsumeKafkaRecord processor for that ? If yes how >>> > do >>> > we specify the dynamic properties ? If possible can you give me an >>> > example >>> > ? >>> > >>> > Regards, >>> > >>> > Chris >> >>
