Hi Dave,
I think you forgot to ask your question :D However, if you're looking to
group those events from the input topic and then putting the aggregated
event onto another topic, have you looked into Kafka Streams?
https://kafka.apache.org/31/documentation/streams/
Cheers,
Liam
On Tue, 3 May 2022 at 12:05, Dave Greeko <davegre...@yahoo.com.invalid>
wrote:
> Hi there,
> We have a number of IoT devices that send multiple related events in
> separate TCP packets to a Kafka broker. The IoT devices have access to a
> PLC system that monitors a set of sensors for an industrial steam boiler.
> The PLC raises multiple independent events for a random number of sensors
> per cycle where each cycle is uniquely identified by an id. Each PLC cycle
> may have a reading from 1 to 10 different sensors each of which is sent
> separately to an IoT device. The IoT device simply relay this event (one
> per sensor) to a Kafka broker. It’s a one-to-many relation (1 PLC cycle
> with a unique id = many events from multiple sensors) our goal is group the
> events per IoT device and plc_cycle_id and the order is constraint by a
> sequence number.
>
> Here is a data sample:
> TCP packet #1:
> {plc_ cycle _id:”AABB”, sensors_per_cycle:3, seq_number:1,
> iot_device_id:1, sensor_data: {name:”pressure”, data:{value:16.3,
> si_unit:”bar”}}
>
> TCP packet #2:
> { plc_ cycle _id:” AABB”, sensors_per_cycle:3, seq_number:2,
> iot_device_id:1, sensor_data: {name:”watter_level”, data:{value:3.8,
> si_unit:”m”}}
>
> TCP packet #3:
> { plc_ cycle _id:” AABB”, sensors_per_cycle:3, seq_number:3,
> iot_device_id:1, sensor_data: {name:”steam_drum_temp”, data:{value:99.6,
> si_unit:”c”}}
>
> Each kafka event has a reference to the total events to be grouped
> (sensors_per_cycle) and the order for which those events will be grouped
> by, is taken from seq_number.
>
> Regards,
> Dave
>
>
