Hello Roman, PFB my response -
As I understand, each protocol has a distinct set of event types (where event type == metrics type); and a distinct set of devices. Is this correct? Yes, correct. distinct events and devices. Each device emits these event. > Based on data protocol I have 4-5 topics. Currently the data for a single event is being pushed to a partition of the kafka topic(producer key -> event_type + data_protocol). Here you are talking about the source (to Flink job), right? Yes, you are right. Can you also share how are you going to consume these data? By consume do you mean the downstream system? If yes then this data will be written to a DB, some metrics goes to TSDB(Influx) as well. Thanks, Hemant On Tue, May 12, 2020 at 2:08 AM Khachatryan Roman < khachatryan.ro...@gmail.com> wrote: > Hi Hemant, > > As I understand, each protocol has a distinct set of event types (where > event type == metrics type); and a distinct set of devices. Is this correct? > > > Based on data protocol I have 4-5 topics. Currently the data for a > single event is being pushed to a partition of the kafka topic(producer key > -> event_type + data_protocol). > Here you are talking about the source (to Flink job), right? > > Can you also share how are you going to consume these data? > > > Regards, > Roman > > > On Mon, May 11, 2020 at 8:57 PM hemant singh <hemant2...@gmail.com> wrote: > >> Hi, >> >> I have different events from a device which constitutes different metrics >> for same device. Each of these event is produced by the device in >> interval of few milli seconds to a minute. >> >> Event1(Device1) -> Stream1 -> Metric 1 >> Event2 (Device1) -> Stream2 -> Metric 2 ... >> .............. >> ....... >> Event100(Device1) -> Stream100 -> Metric100 >> >> The number of events can go up to few 100s for each data protocol and we >> have around 4-5 data protocols. Metrics from different streams makes up a >> records >> like for example from above example for device 1 - >> >> Device1 -> Metric1, Metric 2, Metric15 forms a single record for the >> device. Currently in development phase I am using interval join to achieve >> this, that is to create a record with latest data from different >> streams(events). >> >> Based on data protocol I have 4-5 topics. Currently the data for a single >> event is being pushed to a partition of the kafka topic(producer key -> >> event_type + data_protocol). So essentially one topic is made up of many >> streams. I am filtering on the key to define the streams. >> >> My question is - Is this correct way to stream the data, I had thought of >> maintaining different topic for an event, however in that case number of >> topics could go to few thousands and that is something which becomes little >> challenging to maintain and not sure if kafka handles that well. >> >> I know there are traditional ways to do this like pushing it to >> timeseries db and then joining data for different metric but that is >> something which will never scale, also this processing should be as >> realtime as possible. >> >> Are there better ways to handle this use case or I am on correct path. >> >> Thanks, >> Hemant >> >
