Comparison between jbod vs single large disk for kafka broker

2024-01-10 Thread Jigar Shah
Hello, Currently, I have a capacity issue with my current kafka cluster where in the existing EBS volume is proving insufficient for the new load of incoming data. I need to find a solution to persist more data in my cluster and I see two options - one to vertically scale volume size

Re: Huge latency at consumer side ,testing performance for production and consumption

2022-01-28 Thread Jigar Shah
Hello Liam, Here is the image. I hope it is accessible now *Regards,* *Jigar* On Fri, 28 Jan 2022 at 15:04, Liam Clarke-Hutchinson wrote: > Hi Jigar, > > Your image attachment didn't come through again. > > Thanks, > > Liam > > On Fri, 28 Jan 2022, 5:35 pm Jiga

Re: Huge latency at consumer side ,testing performance for production and consumption

2022-01-27 Thread Jigar Shah
Hello again, Could someone please provide feedback on these findings ? Thank you in advance for feedback. *Regards,* *Jigar* On Mon, 17 Jan 2022 at 13:24, Jigar Shah wrote: > Hello again, > I had performed a few more tests on producer and consumer again and I > observed a pattern

Re: Huge latency at consumer side ,testing performance for production and consumption

2022-01-16 Thread Jigar Shah
analysis, I assume the latency is caused at step *3.a & 3.b *when the partition selected is on the other two brokers. Such peaks are observed during initial part of test only [image: image.png] Thank you in advance for feedback. *Regards,* *Jigar* On Wed, 15 Dec 2021 at 10:53, Jigar Shah w

Re: Configuring topic level configuration MirrorMaker 2.0 in kafka v3.0

2021-12-15 Thread Jigar Shah
e.org/jira/browse/KAFKA-13255>) that we will be > > fixed in v3.2.0. > > I've asked if there is any workaround for this issue in the JIRA > comments. > > > > Thank you. > > Luke > > > > On Wed, Dec 15, 2021 at 1:24 PM Jigar Shah > > wrote: > > > >

Configuring topic level configuration MirrorMaker 2.0 in kafka v3.0

2021-12-14 Thread Jigar Shah
Hello, I am trying to run MirrorMaker 2.0 on kafka version 3.0.0 on my source cluster and my target kafka cluster is in kafka version 2.7.2. I am facing issues with topic configuration to create topics on my target cluster. [2021-12-14 12:28:33,071] WARN [MirrorSourceConnector|worker] Could not

Re: Huge latency at consumer side ,testing performance for production and consumption

2021-12-14 Thread Jigar Shah
> Is there any reason why you need to care about the initial delay? > If, like you said, the delay won't happen later on, I think the cost will > be amortized. > > > Thank you. > Luke > > > On Mon, Dec 13, 2021 at 4:59 PM Jigar Shah > wrote: > > > Hello ,

Re: Huge latency at consumer side ,testing performance for production and consumption

2021-12-13 Thread Jigar Shah
So yeah, this is why you're seeing that initial delay before consuming > records. > > Kind regards, > > Liam Clarke-Hutchinson > > On Mon, 13 Dec 2021, 7:19 pm Liam Clarke-Hutchinson, > wrote: > > > Hi, > > > > I'm assuming you're using consumer groups

Huge latency at consumer side ,testing performance for production and consumption

2021-12-12 Thread Jigar Shah
Hello, I am trying to test the latency between message production and message consumption using Java Kafka-Client*(2.7.2)* library. The configuration of cluster is 3 KafkaBrokers*(2.7.2, Scala 2.13)*, 3 Zookeeper*(3.5.9)* Here is a pattern what I have observed Reference: ConsumerReadTimeStamp:

Error while Consuming message

2015-08-19 Thread Jigar Shah
with correlation id 0. Possible cause: Request for offset 41 but we only have log segments in the range 0 to 38. (kafka.server.ReplicaManager).. TOPICNAME = 12 Total Message are 38 -- [image: photo] Jigar Shah Software Design Engg, Sokrati 9689693323 || dstructures.wordpress.com/ http