Hello We are using Nifi 1.9 with [Consume/Publish]KafkaRecord_2_0 processors and Confluent Content-Encoded Schema Reference... All works fine regarding the payload, but we also use Key (which is also encoded in the same way) As expected if we use different schema or different kafka headers the messages are not placed in the same flow file (as documented and expected), however there is no guarantees for the key, in ConsumeKafka_2_0 the key is part of the flow file attributes and same key messages batch in the same flow file...
I want to make a fork of the [Consume/Publish]KafkaRecord_2_0 and implement the following spec, and want to ask if its consistent with the planned future approach... ConsumeKafkaRecordWithKey_2_0: add property - Key Reader (RecordReader) optional add property - Key Writer (RecordWriter) optional parse the key with reader if set, write key with writer if set FlowFileAttributeWriter (RecordWriter) property - Record Type - Simple (for simple type keys) / Map property - Flow File Attributes Prefix (for my case kafka.key) will take logical record and write as map to flow file attributes PublishKafkaRecordWithKey_2_0 add property - Key Reader (RecordReader) optional add property - Key Writer (RecordWriter) optional if Message Key Field set use as simple key otherwise use Key Reader and Key Writer FlowFileAttributeReader (recordReader) property - Record Type - Simple (for simple type keys) / Map property - Flow File Attributes regex (for my case kafka/.key/..*) property - Flow File Attribute Level Demarcation (default to .) will take flow file attributes and parse as logical record map Is this approach consistent with the guidelines (reader/writer for flow file attributes)? BR, Michael Karpel Data engineering consultant | Matrix BI | +972 54 2680303 | plexa...@gmail.com