Re: Question about applicability of Kafka Streams

[Michael Noll]

Kim,

yes, Kafka Stream is very suitable for this kind of application.

The code example that Tobias linked to should be a good starting point for
you (thanks, Tobias!).

Best,
Michael



On Fri, May 27, 2016 at 4:35 AM, BYEONG-GI KIM  wrote:

> Thank you very much for the information!
> I'll look into it.
>
> Best regards
>
> KIM
>
> 2016-05-27 11:31 GMT+09:00 Tobias Adamson :
>
> > Hi Kim
> > Would maybe this example work for you?
> >
> >
> https://github.com/apache/kafka/tree/trunk/streams/examples/src/main/java/org/apache/kafka/streams/examples/pageview
> > <
> >
> https://github.com/apache/kafka/tree/trunk/streams/examples/src/main/java/org/apache/kafka/streams/examples/pageview
> > >
> >
> > It included JSON -> POJO -> JSON steps and could probably be adapted for
> > your case?
> >
> > Regards
> > Toby
> > > On 27 May 2016, at 10:20 AM, BYEONG-GI KIM  wrote:
> > >
> > > Hello.
> > >
> > > First I thank you so much for the devs since they've been making a
> great
> > > tool as open-source software.
> > >
> > > I'm considering to apply a new feature of the Kafka, aka Kafka Streams,
> > on
> > > my simple handler application, which receives monitoring data from
> > Collectd
> > > and reproduce transformed messages to Kafka Broker(s). For example, I'd
> > > want to change the collected message from Collectd like,
> > >
> > >
> >
> [{"values":[1901474177],"dstypes":["counter"],"dsnames":["value"],"time":1280959128,"interval":10,"host":"
> > > leeloo.octo.it
> > >
> >
> ","plugin":"cpu","plugin_instance":"0","type":"cpu","type_instance":"idle"}]
> > >
> > > to my customized alarm message like,
> > >
> > > {"alarmMsgType":"threshold", "time":145943640, "host":"
> > leeloo.octo.it
> > > ","category":"CPU","type":"IDLE",
> > > "detail":"0","alarmLevel":"critical","message":"cpu
> > > error","value":"1901474177"}
> > >
> > > of course, the re-produced message must be sent to Kafka Broker(s).
> > >
> > > The problem is that, the message(s) from Collectd is Json-formatted so
> > that
> > > it seems the Kafka Streams processing would become complicated, i.e.,
> it
> > > should be JSONParsed from String to JSON and vise versa after
> transform.
> > >
> > > Is it suitable to use the Kafka Stream for this kind of application?
> > >
> > > Any better idea or comments would also really helpful for me. Thanks in
> > > advance!
> > >
> > > Best regards
> > >
> > > KIM
> >
> >
>

Re: Question about applicability of Kafka Streams

[BYEONG-GI KIM]

Thank you very much for the information!
I'll look into it.

Best regards

KIM

2016-05-27 11:31 GMT+09:00 Tobias Adamson :

> Hi Kim
> Would maybe this example work for you?
>
> https://github.com/apache/kafka/tree/trunk/streams/examples/src/main/java/org/apache/kafka/streams/examples/pageview
> <
> https://github.com/apache/kafka/tree/trunk/streams/examples/src/main/java/org/apache/kafka/streams/examples/pageview
> >
>
> It included JSON -> POJO -> JSON steps and could probably be adapted for
> your case?
>
> Regards
> Toby
> > On 27 May 2016, at 10:20 AM, BYEONG-GI KIM  wrote:
> >
> > Hello.
> >
> > First I thank you so much for the devs since they've been making a great
> > tool as open-source software.
> >
> > I'm considering to apply a new feature of the Kafka, aka Kafka Streams,
> on
> > my simple handler application, which receives monitoring data from
> Collectd
> > and reproduce transformed messages to Kafka Broker(s). For example, I'd
> > want to change the collected message from Collectd like,
> >
> >
> [{"values":[1901474177],"dstypes":["counter"],"dsnames":["value"],"time":1280959128,"interval":10,"host":"
> > leeloo.octo.it
> >
> ","plugin":"cpu","plugin_instance":"0","type":"cpu","type_instance":"idle"}]
> >
> > to my customized alarm message like,
> >
> > {"alarmMsgType":"threshold", "time":145943640, "host":"
> leeloo.octo.it
> > ","category":"CPU","type":"IDLE",
> > "detail":"0","alarmLevel":"critical","message":"cpu
> > error","value":"1901474177"}
> >
> > of course, the re-produced message must be sent to Kafka Broker(s).
> >
> > The problem is that, the message(s) from Collectd is Json-formatted so
> that
> > it seems the Kafka Streams processing would become complicated, i.e., it
> > should be JSONParsed from String to JSON and vise versa after transform.
> >
> > Is it suitable to use the Kafka Stream for this kind of application?
> >
> > Any better idea or comments would also really helpful for me. Thanks in
> > advance!
> >
> > Best regards
> >
> > KIM
>
>

Re: Question about applicability of Kafka Streams

[Tobias Adamson]

Hi Kim
Would maybe this example work for you?
https://github.com/apache/kafka/tree/trunk/streams/examples/src/main/java/org/apache/kafka/streams/examples/pageview
 


It included JSON -> POJO -> JSON steps and could probably be adapted for your 
case?

Regards
Toby
> On 27 May 2016, at 10:20 AM, BYEONG-GI KIM  wrote:
> 
> Hello.
> 
> First I thank you so much for the devs since they've been making a great
> tool as open-source software.
> 
> I'm considering to apply a new feature of the Kafka, aka Kafka Streams, on
> my simple handler application, which receives monitoring data from Collectd
> and reproduce transformed messages to Kafka Broker(s). For example, I'd
> want to change the collected message from Collectd like,
> 
> [{"values":[1901474177],"dstypes":["counter"],"dsnames":["value"],"time":1280959128,"interval":10,"host":"
> leeloo.octo.it
> ","plugin":"cpu","plugin_instance":"0","type":"cpu","type_instance":"idle"}]
> 
> to my customized alarm message like,
> 
> {"alarmMsgType":"threshold", "time":145943640, "host":"leeloo.octo.it
> ","category":"CPU","type":"IDLE",
> "detail":"0","alarmLevel":"critical","message":"cpu
> error","value":"1901474177"}
> 
> of course, the re-produced message must be sent to Kafka Broker(s).
> 
> The problem is that, the message(s) from Collectd is Json-formatted so that
> it seems the Kafka Streams processing would become complicated, i.e., it
> should be JSONParsed from String to JSON and vise versa after transform.
> 
> Is it suitable to use the Kafka Stream for this kind of application?
> 
> Any better idea or comments would also really helpful for me. Thanks in
> advance!
> 
> Best regards
> 
> KIM

Question about applicability of Kafka Streams

[BYEONG-GI KIM]

Hello.

First I thank you so much for the devs since they've been making a great
tool as open-source software.

I'm considering to apply a new feature of the Kafka, aka Kafka Streams, on
my simple handler application, which receives monitoring data from Collectd
and reproduce transformed messages to Kafka Broker(s). For example, I'd
want to change the collected message from Collectd like,

[{"values":[1901474177],"dstypes":["counter"],"dsnames":["value"],"time":1280959128,"interval":10,"host":"
leeloo.octo.it
","plugin":"cpu","plugin_instance":"0","type":"cpu","type_instance":"idle"}]

to my customized alarm message like,

{"alarmMsgType":"threshold", "time":145943640, "host":"leeloo.octo.it
","category":"CPU","type":"IDLE",
"detail":"0","alarmLevel":"critical","message":"cpu
error","value":"1901474177"}

of course, the re-produced message must be sent to Kafka Broker(s).

The problem is that, the message(s) from Collectd is Json-formatted so that
it seems the Kafka Streams processing would become complicated, i.e., it
should be JSONParsed from String to JSON and vise versa after transform.

Is it suitable to use the Kafka Stream for this kind of application?

Any better idea or comments would also really helpful for me. Thanks in
advance!

Best regards

KIM