Hi Dave, I think you forgot to ask your question :D However, if you're looking to group those events from the input topic and then putting the aggregated event onto another topic, have you looked into Kafka Streams? https://kafka.apache.org/31/documentation/streams/
Cheers, Liam On Tue, 3 May 2022 at 12:05, Dave Greeko <davegre...@yahoo.com.invalid> wrote: > Hi there, > We have a number of IoT devices that send multiple related events in > separate TCP packets to a Kafka broker. The IoT devices have access to a > PLC system that monitors a set of sensors for an industrial steam boiler. > The PLC raises multiple independent events for a random number of sensors > per cycle where each cycle is uniquely identified by an id. Each PLC cycle > may have a reading from 1 to 10 different sensors each of which is sent > separately to an IoT device. The IoT device simply relay this event (one > per sensor) to a Kafka broker. It’s a one-to-many relation (1 PLC cycle > with a unique id = many events from multiple sensors) our goal is group the > events per IoT device and plc_cycle_id and the order is constraint by a > sequence number. > > Here is a data sample: > TCP packet #1: > {plc_ cycle _id:”AABB”, sensors_per_cycle:3, seq_number:1, > iot_device_id:1, sensor_data: {name:”pressure”, data:{value:16.3, > si_unit:”bar”}} > > TCP packet #2: > { plc_ cycle _id:” AABB”, sensors_per_cycle:3, seq_number:2, > iot_device_id:1, sensor_data: {name:”watter_level”, data:{value:3.8, > si_unit:”m”}} > > TCP packet #3: > { plc_ cycle _id:” AABB”, sensors_per_cycle:3, seq_number:3, > iot_device_id:1, sensor_data: {name:”steam_drum_temp”, data:{value:99.6, > si_unit:”c”}} > > Each kafka event has a reference to the total events to be grouped > (sensors_per_cycle) and the order for which those events will be grouped > by, is taken from seq_number. > > Regards, > Dave > >