Re: Help !! Apache-Kafka On-Premises Setup.

for on-premises setup. If yes then what are recommendations to make it working. Because we are facing issue in name resolution after request landed onto the Kafka broker server as it is not taking the same reverse path to fulfill the request and name resolution as well. Thanks Nitin Sharma On

Re: Help !! Apache-Kafka On-Premises Setup.

e want to know is , How important the DNS is in the Kafka setup while accessing it from open internet and what are the must have requirements to achieve this setup to work in the On-premises environment.* Regards Nitin Sharma . On Fri, Jun 18, 2021 at 7:36 PM Mich Talebzadeh wrote: > Hi, &g

Help !! Apache-Kafka On-Premises Setup.

suggest if it is possible to create the same setup in the Data centre premises instead of Cloud environment. Thanks Nitin Sharma

Kafka Broker : Socket.request.max.bytes & Socket.receive.buffer.bytes

Hi All, I am trying to increase my Kafka producer client throughput and for same, i was exploring server.properties file parameters. I would like to better understand the relation between "Socket.request.max.bytes" and "Socket.receive.buffer.bytes" parameter sets in broker's server.properties fil

What it means by Committed logs

Hi Team, I have couple of questions related to Log commit : a) What Committed Logs means? i understand that Consumer can pick messages when it is committed, i.e. when message reached to all in-sync replicas. But it leaves doubt in my mind. does commit means when all replicas have flushed messa

Re: Kafka AdminClient

ent yet. It is something > we would like to support in the future. > > Ismael > > On Mon, Jul 17, 2017 at 12:16 AM, nitin sharma < > kumarsharma.ni...@gmail.com> > wrote: > > > Hi All, > > > > My requirement is to automate "Partition Rebalan

Kafka AdminClient

Hi All, My requirement is to automate "Partition Rebalance" through java APIs, rather than .sh scripts. I am in assumption that Kafka AdminClient (0.11.0 v) have these features and hence kindly let me know if it is TRUE. Also, if AdminCLient API are document on net then kindly guide me to the l

Re: Delay in Zookeeper Offset updates

of consumers for a given cluster, we actually had to increase > the interval to 10 minutes for quite some time because the load on > Zookeeper was just too high. Once we moved the ZK transaction logs to SSDs, > we were able to drop that back to 60 seconds. > > -Todd > > > >

Re: How Kafka Distribute partitions across the cluster

er options as well(KAFKA-2435 > <https://issues.apache.org/jira/browse/KAFKA-2435>, KAFKA-1215 > <https://issues.apache.org/jira/browse/KAFKA-1215>, KAFKA-1736 > <https://issues.apache.org/jira/browse/KAFKA-1736>). > > Thanks, > Grant > > On Tue, Oct 27, 2

How Kafka Distribute partitions across the cluster

Hi All, I would like to know what kind of strategy Kafka adopts in splitting partitions of a topic to across the cluster. Ex: if i have 10 broker in a cluster. 1 topic with 5 partition and each with replication-factor of 2. then how would Kafka will assign the partitions ( 5*2 =10) across the clu

Re: Delay in Zookeeper Offset updates

anyone faced this issue? Regards, Nitin Kumar Sharma. On Wed, Sep 2, 2015 at 5:32 PM, nitin sharma wrote: > Hi All, > > i have run into a weird issue with my Kafka setup. > > I see that it takes around 5-6 sec for Zookeeper to update the offset for > Kafka topics

Delay in Zookeeper Offset updates

Hi All, i have run into a weird issue with my Kafka setup. I see that it takes around 5-6 sec for Zookeeper to update the offset for Kafka topics. I am running "ConsumerOffsetChecker" tool to see that lag and what i found that even when my consumer is not up, it takes 5-6 sec for Zookeeper to sh

Usage of Kafka Mirror Maker

Hi Team, I would like to know under which all circumstances I can use MirrorMaker process ? i read on a website that : "*it is not really intended as a fault-tolerance mechanism*" , so curious to know where all i can think of using MirrorMaker. Also, how to know the version of MirrorMaker proces

Re: Kafka server relocation

y need to tune it > > On Mon, Apr 13, 2015 at 12:43 PM, nitin sharma < > kumarsharma.ni...@gmail.com> > wrote: > > > hi Xiao, > > > > i have finally got JMX monitoring enabled for my kafka nodes in test > > envrionment and here is what i observed.

Re: Kafka server relocation

egards, Nitin Kumar Sharma. On Wed, Apr 8, 2015 at 10:48 PM, tao xiao wrote: > Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to check if > the consumer has problem processing messages > > On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma > wrote: > > > thank

Re: Kafka server relocation

cause. metrics in kafka.consumer and kafka.producer > categories will help you find out the problems. > > This link gives instruction how to read the metrics > http://kafka.apache.org/documentation.html#monitoring > > > On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma > wrot

Re: Kafka server relocation

r maker log? > > On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma > > wrote: > > > i tried with auto.offset.reset=smallest, but still not working.. > > > > there is data in my source cluster > > > > Regards, > > Nitin Kumar Sharma. > > > > &

Re: Kafka server relocation

On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma > > wrote: > > > thanks Xiao > > > > I tried MirrorMaker option in my test environment but failed. I am not > > able to see the log getting copied to destination cluster. I see in the > log > > of MirrorM

Re: Kafka server relocation

be able to consume > data from N sources (N >= 1) and transfer data to one destination cluster. > You are free to supply as many consumer properties as you want to instruct > mirror maker where to consumer data from. > > On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma > > wrote:

Re: Kafka server relocation

er and add them to > > the old cluster. > > 2. Use replica assignment tool to reassign all the partitions to brokers > > in new datacenter. > > 3. Perform controlled shutdown on the brokers in old datacenter. > > > > Jiangjie (Becket) Qin > > > > O

Kafka server relocation

Hi Team, in my project, we have built a new datacenter for Kafka brokers and wants to migrate from current datacenter to new one. Switching producers and consumers wont be a problem provided New Datacenter has all the messages of existing Datacenter. i have only 1 topic with 2 partition that ne

Re: Kafka Leader Rebalance

and are you using the kafka-topics tool to > see which leaders are on which brokers? > Did you also wait for more than 15 seconds to see if the leader election > takes place? > Are there any errors in the controller log? > > On Sat, Feb 14, 2015 at 9:35 AM, nitin sharma > >

Re: Kafka Leader Rebalance

Hi, Anyone who knows how to solve my problem, please respond.. Regards, Nitin Kumar Sharma. On Thu, Feb 12, 2015 at 3:50 PM, nitin sharma wrote: > Hi Team, > > i need some help in solving my current issue related to > "kafka-leadership-rebalance" > > I have 2

Kafka Leader Rebalance

Hi Team, i need some help in solving my current issue related to "kafka-leadership-rebalance" I have 2 brokers.. i have deployed 2 topics with 2 partition and 2 replica each in following format.. I made use of kafka-reassignment.sh for same Topic partition Leader Fol

Kafka-Consumer-perf.test.sh -- how to check if it is running

Hi All, I am running kafka-consumer-perf.test.sh in my test envrionment to simulate "consumer" load on my kafka broker. Currently i have millions of entries in the log on which this shell is running... I would like to know how can if check if the .sh is running fine.. All i see is the below en

Kafka under replicated partitions..

Hi All, I would like to know the factors that can cause followers to fall behind and under-replicated replica getting created? In my production system , last Friday one of the partitions fell behind and before i could check my operation team had restarted the follower broker... Restart did fixed

Re: kafka deleted old logs but not released

hi All, I am facing the same issue.. i have got Kafka_2.9.2-0.8.1.1.jar deployed in my application. My Operation team had reported significant increase in the directory where Kafak-log folder exists . when i checked the size of parent directory is it 99GB (100%) utilized.. This was very confusin

Re: kafka-producer-perf-test.sh

knows the limitation of this utility, so that i can think of writing my own in order to overcome the limitations. Regards, Nitin Kumar Sharma. On Mon, Dec 22, 2014 at 10:57 AM, Jun Rao wrote: > What kind of load testing do you plan to do? > > Thanks, > > Jun > > On Wed,

Re: Produce 1 million events/seconds

Hey Pramod, few things: a. You can keep 2 brokers but you can increase the ProducerSend thread on your producer side to push more messages. best way try to create more threads that execute the loop where "send" is called. b. try to avoid/reduce putting any logic computation in the while loop .. Yo

Re: Produce 1 million events/seconds

btw -- curious to know how well your kafka broker handles the load... please do inform us your result. Regards, Nitin Kumar Sharma. On Mon, Dec 22, 2014 at 9:52 AM, nitin sharma wrote: > Hey Pramod, > > few things: > a. You can keep 2 brokers but you can increase the ProducerSe

Re: Max. storage for Kafka and impact

hi, Few things you have to plan for: a. Ensure that from resilience point of view, you are having sufficient follower brokers for your partitions. b. In my testing of kafka (50TB/week) so far, haven't seen much issue with CPU utilization or memory. I had 24 CPU and 32GB RAM. c. 200,000 partitions

Re: Increase in Kafka replication fetcher thread not reducing log replication

HI All -- has any one faced the problem i have posted in my previous email ? Regards, Nitin Kumar Sharma. On Wed, Dec 17, 2014 at 11:08 AM, nitin sharma wrote: > > Hi All, > > I am trying to figure out best configuration for my Kafka brokers so that > in case of restarted, the

Increase in Kafka replication fetcher thread not reducing log replication

Hi All, I am trying to figure out best configuration for my Kafka brokers so that in case of restarted, the new node catch up with Leader at quick pace. My test environment has 2 kafka brokers and 1 Topic with one Partition. I first ran the test (Test#1) with default setting, i.e. num.replica.fe

kafka-producer-perf-test.sh

Hi Team, I have to make a decision on whether i should go with Kafka producer test utility or build my own java tool for my load testing . Kindly let me know if anyone knows any limitation with "kafka-producer-per-test.sh" when it come to simulation of messages under load condition? Regards, Nit

Re: Number of Consumers Connected

got it ... thanks a lot. Regards, Nitin Kumar Sharma. On Mon, Dec 15, 2014 at 9:26 PM, Gwen Shapira wrote: > > Hi Nitin, > > Go to where you installed zookeeper and run: > > bin/zkCli.sh -server 127.0.0.1:2181 > > On Mon, Dec 15, 2014 at 6:09 PM, nitin sharma > wrot

Re: Number of Consumers Connected

> ls /consumers > > > > and count the number of results > > > > On Mon, Dec 15, 2014 at 11:34 AM, nitin sharma > > wrote: > > > Hi Team, > > > > > > Is it possible to know how many Consumer Group connected to kafka > broker > > Ids &

Number of Consumers Connected

Hi Team, Is it possible to know how many Consumer Group connected to kafka broker Ids and as well as how many Instances within a Group are fetching messages from Kafka Brokers Regards, Nitin Kumar Sharma.

Re: How to raise a question in forum

values coming in negative. Regards, Nitin Kumar Sharma. On Wed, Dec 10, 2014 at 4:34 PM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi, > > On Wed, Dec 10, 2014 at 12:02 AM, nitin sharma < > kumarsharma.ni...@gmail.com> > wrote: > > > thanks for r

Re: How to Ingest data into kafka

Hi Kishore, You can use Kafka Producer API for same. You can find the sample code on Kafka Quick Start page : http://kafka.apache.org/07/quickstart.html Regards, Nitin Kumar Sharma. On Wed, Dec 10, 2014 at 2:14 AM, kishore kumar wrote: > Hi kafkars, > > I want to write a java code to ingest

Re: How to raise a question in forum

a few times before: > http://search-hadoop.com/?q=monitor+lag&fc_project=Kafka > > Otis > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support * http://sematext.com/ > > > On Tue, Dec 9, 2014 at 4:59 PM, nitin sha

Re: How to raise a question in forum

ox/%3cca+ttwkmbjhy_q+io3_7xsqxxfc6hhf7pzap5sfgtzd22stb...@mail.gmail.com%3E Regards, Nitin Kumar Sharma. On Tue, Dec 9, 2014 at 4:47 PM, Guozhang Wang wrote: > Nitin, > > You have found the place :) Just send an email to users@ mailing list is > fine. > > Guozhang > >

How to raise a question in forum

HI, I need some help related to Kafka Consumer Lag monitoring.. Kindly let me know how can i post my query to the forum. Regards, Nitin Kumar Sharma.