Re: Advice for Kafka project in Africa...

2020-09-03 Thread Gaurav Bajaj
Hello Cedric, My 2 cents : We use Kafka alot but mostly for messaging and event streaming purpose. Using Kafka as Database is not a usecase i would look Kafka for. Ofcourse you can use it to store some intermediate states but having it as "system of records" would be stretched use case for Kafka.

Re: Consumer Group rebalancing events

2020-09-03 Thread Devaki, Srinivas
Hi Tess D'erberwill, Таисия Дорошенков, Thanks a lot for the reply on this, one major issue I found with our existing metrics is that they are not granular enough i.e if the rebalancing finishes under 10 seconds then I might lose that event in the metrics if my metrics scrape interval is 10 second

Re: Kafka streams parallelism - why not separate stream task per partition per input topic

2020-09-03 Thread Matthias J. Sax
Well, it depends on your program. The reason for the current task creating strategy are joins: If you have two input topic that you want to join, the join happens on a per-partition basis, ie, topic-p0 is joined to topicB-p0 etc and thus both partitions must be assigned to the same task (to get co

Re: Streams constantly reblancing,

2020-09-03 Thread Steve Jones
There are zero events on the stream in the case with the log files, I've had the same behaviour when running with 6 events. Its constantly reporting rebalancing but from a CPU perspective nothing appears to actually be doing anything. I've run it for several hours just to see if something happene

Re: UPGRADING ZOOKEEPER FROM 3.4.13 TO 3.5.7

2020-09-03 Thread Manoj.Agrawal2
Issue has been fixed by copying empty snapshot file to data dir . Thanks . On 9/2/20, 10:51 PM, "Enrico Olivelli" wrote: [External] The official way to fix it is here https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FZOOKEEPER-

Re: Kafka compatibility with ZK

2020-09-03 Thread Andrey Klochkov
Hello all, FWIW we upgraded to Kafka 2.4.1 and kept ZK at 3.4.6, no issues noticed. On Sun, Aug 2, 2020 at 10:04 AM Marina Popova wrote: > > Actually, I'm very interested in your experience as well I'm about to > start the same (similar) upgrade - from Kafka 0.11/ZK3.4.13 to Kafka 2.4/ZK > 3

Re: Kafka compatibility with ZK

2020-09-03 Thread Manoj.Agrawal2
We also upgraded kafka 2.2.1 to kafka 2.5.0 and kept same zookeeper . no issued reported . Later we also upgraded zookeeper to 3.5.8 . all good . On 9/3/20, 8:42 PM, "Andrey Klochkov" wrote: [External] Hello all, FWIW we upgraded to Kafka 2.4.1 and kept ZK at 3.4.6, no issues not