Re: KStreams regex-topic null pointer Exception with multiple Consumer processes

2016-09-04 Thread Guozhang Wang
Hello David, Thanks for reporting this issue, and after some look through the code I think it is indeed a bug, and commit 6fb33afff976e467bfa8e0b29eb82770a2a3aaec will not fix it IMHO. Would you want to create a JIRA for keeping track of this issue? Guozhang On Sat, Sep 3, 2016 at 10:16 AM,

Re: Monitoring the max lag of a kafka streams application.

2016-09-04 Thread Guozhang Wang
Hi Rohit, Since "records-lag-max" is defined as the Max stat, it is initialized as -infinity, and will be updated only as "max(current-value, watermark - last fetched offset)" when the fetcher does return some data. If you always see "-infinity" then it seems the job started at the end offsets

Using kafka as a "message bus" for an event store

2016-09-04 Thread F21
Hi all, I am currently looking at using Kafka as a "message bus" for an event store. I plan to have all my events written into HBase for permanent storage and then have a reader/writer that reads from HBase to push them into kafka. In terms of kafka, I plan to set it to keep all messages

Brokers / Best practices to set log.flush.interval.*

2016-09-04 Thread Florian Hussonnois
Hi all, I would like to know how to configure the following paramaters : log.flush.interval.messages log.flush.interval.ms log.flush.scheduler.interval.ms The Kafka 0.8.X documentation indicates it is not recommanded to set these parameters as this can have major impact on performance. But