Re: Kafka Streams Session store performance degradation from 0.10.2.1 to 0.11.0.3

2019-01-11 Thread Guozhang Wang
Hello Jonathan, I've left a comment on https://issues.apache.org/jira/browse/KAFKA-7652 with a fix trying to resolve the discovered bug in trunk. If it verifies to be the right fix I will push it to older branches as well. Just FYI. Guozhang On Fri, Nov 16, 2018 at 4:26 PM Guozhang Wang wrote

Re: Streams SessionStore - bug or documentation error?

2019-01-11 Thread Guozhang Wang
Hello Tom, This is a long-delayed reply, but I've finally discovered the bug you reported inside session stores, and here is a fix that I've put up for trunk: https://github.com/apache/kafka/pull/6134 Sorry it took a while for me to get back to you! Guozhang On Wed, Oct 17, 2018 at 8:31 AM Gu

Degraded system throughput with time with no change in network usage

2019-01-11 Thread Prajwal
Hey there, I'm trying build a data ingestion pipeline on azure. My resources: 1 * Producer [having zookeeper installed with it] (16 Gb Ram and 4 Cores) 2 * Brokers (16Gb Ram and 4Cores) I'm attempting to transfer log files which have variable size ranging from 0.1 Mb to 0.7 Mb. Current having 2

Re: Configuration guidelines for a specific use-case

2019-01-11 Thread Ryanne Dolan
Given your high throughput on the consumer side, you might consider adding more partitions than 6, so you can scale up beyond 6 consumers per group if need be. Ryanne On Wed, Jan 9, 2019 at 10:37 AM Gioacchino Vino wrote: > Hi Ryanne, > > > I just forgot to insert the "linger.ms=0" configuratio

Re: Aw: Re: Doubts in Kafka

2019-01-11 Thread Pulkit Manchanda
Aruna, Instead of using 1000 partitions you can chose to create dynamic topics and have a consumer for each topic. On Fri, Jan 11, 2019 at 6:43 AM Peter Levart wrote: > > > On 1/10/19 2:26 PM, Sven Ludwig wrote: > > Okay, but > > > > what if one also needs to preserve the order of messages comin

Re: Aw: Re: Doubts in Kafka

2019-01-11 Thread Peter Levart
On 1/10/19 2:26 PM, Sven Ludwig wrote: Okay, but what if one also needs to preserve the order of messages coming from a particular device? With Kafka, this is perhaps possible if all messages from a particular device go into the same partition. Would it be a good and efficient solution