Re: Stream processing meetup at LinkedIn (Mountain View) on Tuesday, August 23 at 6pm

asking! > > Ed > > On Fri, Aug 12, 2016 at 2:56 PM, Prabhjot Bharaj > wrote: > > > Hi, > > > > Thanks for the invitation. I won't be able to make it this soon. > > However, it'll be great if you could arrange to share the video > recordings. >

Re: Stream processing meetup at LinkedIn (Mountain View) on Tuesday, August 23 at 6pm

Hi, Thanks for the invitation. I won't be able to make it this soon. However, it'll be great if you could arrange to share the video recordings. Thanks, Prabhjot On Aug 12, 2016 4:33 PM, "Joel Koshy" wrote: > Hi everyone, > > We would like to invite you to a Stream Processing Meetup at > Linke

Re: SV: What is the benefit of using acks=all and minover e.g. acks=3

6 > acks=all > retries=0 > timeout.ms=3 > buffer.memory=67108864 > metadata.fetch.timeout.ms=3000 > > Do let me know what you think about it! We are currently setting up some > tests using the broker properties that you suggested. > > Regards > Andreas > &g

Re: What is the benefit of using acks=all and minover e.g. acks=3

ioritize when things get clogged for some reason? > > Sorry for the quite messy description, we are all kind of new at kafka > here! > > BR > Andreas > > > On 28 Nov 2015, at 09:26, Prabhjot Bharaj wrote: > > > > Hi, > > > > This should help :) >

Re: What is the benefit of using acks=all and minover e.g. acks=3

s will fall behind, you will have 3 out of 4 in sync. > If you set min.insync.replica = 3, produce requests will fail if the number > on in-sync replicas fall below 3. > > I hope this helps. > > Gwen > > On Fri, Nov 27, 2015 at 9:43 PM, Prabhjot Bharaj > wrote: > >

Re: What is the benefit of using acks=all and minover e.g. acks=3

Hi Gwen, How about min.isr.replicas property? Is it still valid in the new version 0.9 ? We could get 3 out of 4 replicas in sync if we set it's value to 3. Correct? Thanks, Prabhjot On Nov 28, 2015 10:20 AM, "Gwen Shapira" wrote: > In your scenario, you are receiving acks from 3 replicas whil

Re: Notable failure scenarios in high-level/new consumer

Hello Folks, Request your expertise on these doubts of mine Thanks, Prabhjot On Thu, Nov 26, 2015 at 11:28 PM, Prabhjot Bharaj wrote: > Hi, > > Request your expertise on these doubts of mine > > Thanks, > Prabhjot > > On Thu, Nov 26, 2015 at 12:09 PM, Prabhjot Bha

Re: producer-consumer issues during deployments

you will be able to consumer from it. If the leader is on a > machine that goes offline then consumption will pause whilst leadership > switches over to a replica. > > All the best > B > > > On 26 Nov 2015, at 17:58, Prabhjot Bharaj wrote: > > > > Hi, > > &g

Re: Notable failure scenarios in high-level/new consumer

Hi, Request your expertise on these doubts of mine Thanks, Prabhjot On Thu, Nov 26, 2015 at 12:09 PM, Prabhjot Bharaj wrote: > Hello Folks, > > I am trying to build fault tolerance on the consumer side, so as to make > sure that all failure scenarios are handled. > On Data

Re: producer-consumer issues during deployments

Hi, Request your expertise on these doubts of mine Thanks, Prabhjot On Thu, Nov 26, 2015 at 4:43 PM, Prabhjot Bharaj wrote: > Hi, > > We arrange our kafka machines in groups and deploy these phases. > > For kafka, we’ll have to map groups with phases. During each phase of the

producer-consumer issues during deployments

Hi, We arrange our kafka machines in groups and deploy these phases. For kafka, we’ll have to map groups with phases. During each phase of the release, all the machines in that group can go down. When this happens, there are a couple of cases:- 1. All replicas are residing in a group of mach

Notable failure scenarios in high-level/new consumer

Hello Folks, I am trying to build fault tolerance on the consumer side, so as to make sure that all failure scenarios are handled. On Data integrity side, there are primary 2 requirements:- 1. No Data loss 2. No data duplication I'm particularly interested in data duplication. e.g. there are var

Re: kafka 0.8 producer issue

Hi, >From the information that you've provided, I think your callback is the culprit here. It is seen from the stacktrace:- at com.hpe.ssmamp.kafka.KafkaPDFAProducer$1.onCompletion( KafkaPDFAProducer.java:62) Please provide more information like a code snippet etc, so that we can tell more

Re: kafka producer and client use same groupid

Hi Surender, Please elaborate on your design Consumers don't talk to producers directly, Kafka is a brokered system, and Kafka sits between producers and consumers Also, consumers consume from partitions of a topic and producers write to partitions in a topic These partitions and the logical abstr

Re: All brokers are running but some partitions' leader is -1

Hi, With the information provided, these are the steps I can think of (based on the experience I had with kafka):- 1. do a describe on the topic. See if the partitions and replicas are evenly distributed amongst all. If not, you might want to try the 'Reassign Partitions Tool' - https://cwiki.apa

Re: Subscription to the mailing list

Hi, You can subscribe using the steps mentioned here: http:// kafka.apache.org / contact.html Regards, Prabhjot On Nov 23, 2015 8:51 PM, "Devill

Re: replication factor increased automatically !

Hi Sunil, Please paste the full output, including the first line of the output of kafka-topics.sh. e.g. root@x.x.x.x:~# kafka-topics.sh --describe --zookeeper localhost:2182 --topic part_2_repl_3 *Topic:part_2_repl_3 PartitionCount:2 ReplicationFactor:3 Configs:* Topic: part_2_repl_3 Partition

Re: consumer offset tool and JMX metrics do not match

; I believe producers / brokers / and consumers has been restarted at > different times. > What do you think the issue is? > > On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj > wrote: > > > Hi, > > > > Did any of your brokers/consumers undergo a restart between t

Re: Subscription Request

Hi Saurabh, Sure thing. You can follow the steps here: http://kafka.apache.org/contact.html Regards, Prabhjot On Nov 18, 2015 11:32 AM, "saurabh pratap singh" wrote: > Hi > > I have recently started using kafka and I want to subscribe to mailing list > . > > > Thanks > Saurabh >

Re: consumer offset tool and JMX metrics do not match

Hi, Did any of your brokers/consumers undergo a restart between the time after you had started the consumption and until you see this issue ? Thanks, Prabhjot On Sat, Nov 14, 2015 at 5:53 AM, allen chan wrote: > I also looked at this metric in JMX and it is also 0 > *kafka.consumer:type=Cons

Re: Kafka Scalability & Partition

Hi, Having such a model will not scale. I think it it's mentioned in earlier posts as well as some wikis available out there Kafka works very well if you have more partitions, so you can massively parallel writes to Kafka. Also, your application need not send partition ids/numbers, unless you've

Re: Re: Re: Re: Re: Kafka lost data issue

Hi Gwen, If producer cant get a response but the message got committed, Because of retries, can this producer be committing messages more than once?? I'm trying to see this problem from the point of view of Idempotent Producer Thanks, Prabhjot On Nov 13, 2015 1:26 PM, "Hawin Jiang" wrote: > Tha

Re: Re: Kafka lost data issue

Hi, Just to confirm that the number of messages produced are what you are seeing, What does GetOffsetShell report for this topic ? Regards, Prabhjot On Thu, Nov 12, 2015 at 2:13 PM, Hawin Jiang wrote: > Hi Jinxing > > I don't think we can resolve this issue by increasing producers. if I > in

Re: Locality question

Hi Ben, I have a similar use case where data from processing layer gets onto storage layer, where data per db gets distributed amongst storage machines based on some logic. The architecture that we had agreed upon was to have topics as storage machine names so the processing layer can send all dat

Re: Debug kafka code in intellij

t: /Users/gwen/workspaces/kafka/clients/build/classes/main > Test output: /Users/gwen/workspaces/kafka/clients/build/classes/test > > Make sure you have something similar. > > Gwen > > On Thu, Nov 5, 2015 at 9:24 PM, Prabhjot Bharaj > wrote: > > > Hi Gwen, > >

Re: Debug kafka code in intellij

asses if needed. Regards, Prabhjot On Fri, Nov 6, 2015 at 10:00 AM, Gwen Shapira wrote: > I also have a gradle plugin, but I don't think its used for debugging. > > What prevents you from debugging? What happens when you right-click on a > test and select "debug"? > >

Re: Debug kafka code in intellij

mailto: > > > > > ra...@gruchalski.com (mailto:ra...@gruchalski.com)) > > > > > de.linkedin.com/in/radgruchalski/ ( > > http://de.linkedin.com/in/radgruchalski/) ( > > > > > http://de.linkedin.com/in/radgruchalski/) > > > > > > >

Re: Debug kafka code in intellij

Hello Folks, Requesting your expertise on this Thanks, Prabhjot On Thu, Nov 5, 2015 at 4:46 PM, Prabhjot Bharaj wrote: > Hi, > > I'm using kafka 0.8.2.1 version with IntelliJ. > Sometimes, I change the code and build it using this command: > > ./gradlew -PscalaVers

Re: New and updated producers and consumers

t fully supports security features like SSL > > > that are slated to be released into the platform. > > > > > > Your choice of development language is entirely up to you. Note that > the > > > only version of clients that will be maintained in the

Known issues or points of failure during consumer rebalance

Hello Folks, I am evaluating some failure scenarios during consumer rebalance in the high-level consumer. The idea of this test is to know what are the pain points from operational/maintanence stand poitn that I need to consider when a consumer rebalance takes place. Also, if there are any known

Re: New and updated producers and consumers

as simple) Thanks, Prabhjot Thanks, Prabhjot On Thu, Nov 5, 2015 at 3:55 PM, Prabhjot Bharaj wrote: > Adding users as well > > On Thu, Nov 5, 2015 at 3:37 PM, Prabhjot Bharaj > wrote: > >> Hi, >> >> I'm using the latest update: 0.8.2.2 >> I would

Debug kafka code in intellij

Hi, I'm using kafka 0.8.2.1 version with IntelliJ. Sometimes, I change the code and build it using this command: ./gradlew -PscalaVersion=2.11.7 releaseTarGz In some cases, I feel the need for debugging the code within IntelliJ. e.g. I, currently, want to debug the ConsumerOffsetCheker to see h

Re: Controlled shutdown

Hi Vadim, Did you see your hard disk partition getting full where kafka data directory is present ? It could be because you have set log retention to a larger value, whereas your input data may be taking up full disk space. In that case, move some data out from that disk partition, set log retenti

Re: New and updated producers and consumers

Adding users as well On Thu, Nov 5, 2015 at 3:37 PM, Prabhjot Bharaj wrote: > Hi, > > I'm using the latest update: 0.8.2.2 > I would like to use the latest producer and consumer apis > over the past few weeks, I have tried to do some performance benchmarking > using th

Re: Kafka Out Of Memory Error

y rule of > thumb regarding this matter? Can you share your configuration (including > spec, jvm memory, etc) for kafka cluster? > > Thank you, > > On Thu, Nov 5, 2015 at 4:07 PM, Prabhjot Bharaj > wrote: > > > Hi, > > > > Not sure. But, I had hit OOM when us

Re: Kafka Out Of Memory Error

Hi, Not sure. But, I had hit OOM when using too many partitions and many topics with a lesser heap size assigned to the jvm Regards, Prabhjot On Thu, Nov 5, 2015 at 10:40 AM, Muqtafi Akhmad wrote: > Hello all, > Recently I got incident with kafka cluster, I found OutOfMemoryError in > kafka se

Re: Kafka console producer stuck

Hi, It seems (from VerifiableProperties.scala - https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/utils/VerifiableProperties.scala#L224) that you are providing some property which kafka does not recognise Please share the full command that you are trying to use Regards, Prabhjo

Re: Queue Full

Hi, This is a type of problem where you operating more than network capacity. This can be handled at two places (you decide whichever is useful/practical for use case) :- 1. In case the bottleneck is because of the broker slowness, increase the number of partitions of your topic, balance them ou

Re: Urban Airship Connect

le replication. Most of the time end-to-end is less than > 500ms, but that's subject to the latency of the decoration services. > > On Fri, Oct 9, 2015 at 1:23 PM, Prabhjot Bharaj > wrote: > > > Hi, > > > > Nice to know another usage of Kafka > > > &

Re: Is kafka 8 and LeaderNotAvailableError

Wow. Nice. So, what all did you change in your cluster?? Regards, Prabhjot On Oct 11, 2015 7:58 PM, "David Montgomery" wrote: > Its working great now. Thanks > > On Sat, Oct 10, 2015 at 3:35 AM, Prabhjot Bharaj > wrote: > > > Zoo.cfg looks OK to me > &g

Re: Urban Airship Connect

Hi, Nice to know another usage of Kafka Could you tell how much load does the first and second level Kafka clusters intake? What is your write throughput, and end-to-end latency, for how long do you hold data in Kafka and how many minimum in-sync replicas you've chosen? Thanks, Prabhjot On Oct

Re: Is kafka 8 and LeaderNotAvailableError

lientPort=2181 > > server.1=45.55.xxx.xxx:2888:3888 > server.3=104.236.xxx.xxx:2888:3888 > server.2=192.241.xxx.xxx:2888:3888 > > > On Sat, Oct 10, 2015 at 3:02 AM, Prabhjot Bharaj > wrote: > > > Hi, > > > > The error is due to some kind of timeout hap

Re: Is kafka 8 and LeaderNotAvailableError

t;> read 42 bytes from Kafka > >> 2015-10-10 > >> 00:17:38,143.143.234968185:kafka.conn:140388282590976:DEBUG:20143:Read > >> 42/42 bytes from Kafka > >> 2015-10-10 > >> > 00:17:38,143.143.313884735:kafka.client:140388282590976:DEBUG:20143:Response > &

Re: Is kafka 8 and LeaderNotAvailableError

ot;controller_epoch":10,"leader":26665,"version":1,"leader_epoch":2,"isr":[26665]} > >> > >> brokers/topics/topic-test-production/partisions/1/state > {"controller_epoch":10,"leader":26665,"version":1,"leader_epoch"

Re: Is kafka 8 and LeaderNotAvailableError

unt:2 ReplicationFactor:2 Configs: > > Topic: topic-test-production Partition: 0 Leader: 26665 Replicas: > > 26665,2028 Isr: 26665 > > Topic: topic-test-production Partition: 1 Leader: 26665 Replicas: > > 2028,26665 Isr: 26665 > > > > But in kafka

Re: Is kafka 8 and LeaderNotAvailableError

Hi, Is this a valid zookeeper connection string 1.zk.do.production.test:2181,2.zk.do.production.test:2181,3.zk.do.production.test:2181 Is 2.zk.do.production.test or 3.zk.do.production.test a valid hostname?? Regards, Prabhjot On Oct 9, 2015 1:37 PM, "David Montgomery" wrote: > PS I added one

Re: Producer becomes slow over time

machine that is not part of the kafka cluster? > > Can you include your topic description/configuration, producer > configuration, and broker configuration? > > On 9/24/15, 1:44 AM, "Prabhjot Bharaj" wrote: > > >Hi, > > > >I would like to dig deep into t

Re: using kafka

Hi, If you put n different consumers in different consumer groups, each consumer will get the same message. Each consumer gets full data But, if you put n consumers in 1 consumer group, it'll act as a traditional distributed queue. Amortised, each consumer will get 1/n of the overall data Regard

Re: Producer becomes slow over time

46 PM, Prabhjot Bharaj wrote: > Hello Folks, > > I've noticed that 2 producer machines, that I had configured, have become > very slow over time > They are giving 17-19 MB/s > > But, a producer that I setup today is giving 70MB/s as the write throughput > > If I

Producer becomes slow over time

Hello Folks, I've noticed that 2 producer machines, that I had configured, have become very slow over time They are giving 17-19 MB/s But, a producer that I setup today is giving 70MB/s as the write throughput If I see the contents of bin, libs, config directories, nothing is different in the fi

Re: Useful metric to check slow ISR catchup

created a new topic with 1 partition and 3 replicas, I see similar graph on the other set of machines. I'm not sure what parameter could be causing this. Any pointers are appreciated Thanks, Prabhjot On Mon, Sep 21, 2015 at 1:20 PM, Prabhjot Bharaj wrote: > Hello Folks, > > Request you

Re: Will Mirror Maker only support 1 to 1?

Hi, Can I have 2 separate mirror maker processes in this format:- Process 1 - source: 2, target: 1 Process 2 - source: 3, target: 1 If this is not supported, then will this circular kind of setup work ? Process 1 (on cluster 1) - source: 2, target: 1 Process 1 (on cluster 2) - source: 1, target

Re: Useful metric to check slow ISR catchup

Hello Folks, Request your expertise on this Thanks, Prabhjot On Fri, Sep 18, 2015 at 6:18 PM, Prabhjot Bharaj wrote: > Hi, > > I've noticed that 1 follower replica node out of my kafka cluster catches > up to the data form the leader pretty slowly. > My topic has jus

Useful metric to check slow ISR catchup

Hi, I've noticed that 1 follower replica node out of my kafka cluster catches up to the data form the leader pretty slowly. My topic has just 1 partition with 3 replicas. One other follower replica gets the full data from the leader pretty instantly It takes around 22 minutes to catch up 500MB of

Re: Jumbled up ISR

icaLeaderElectionTool > > > Thank you, > Grant > > On Tue, Sep 15, 2015 at 5:26 AM, Prabhjot Bharaj > wrote: > > > Hi, > > > > Topic:part_1_repl_3_3 PartitionCount:1 ReplicationFactor:3 Configs: > > > > Topic: part_1_repl_3_3 Partition: 0 Lead

Jumbled up ISR

Hi, Topic:part_1_repl_3_3 PartitionCount:1 ReplicationFactor:3 Configs: Topic: part_1_repl_3_3 Partition: 0 Leader: 3 Replicas: 3,4,5 Isr: 4,3,5 I see that the replicas are 3,4,5 but, the ISR is 4,3,5. I have this doubt:- When the leader is 3, can the ISR be 4, 3, 5 ? Does the ISR got to ha

Re: What can be reason for fetcher thread for slow response.

Hi, In addition to the parameters asked by Erik, it would be great if you could share your broker's server.properties as well Regards, Prabhjot On Fri, Sep 11, 2015 at 8:10 PM, Helleren, Erik wrote: > Hi Madhukar, > Some questions that can help understand whats going on: Which kafka > version

Re: Slow ISR catch-up

Hello friends, Request your expertise on this problem I'm facing Thanks On Sep 4, 2015 8:09 PM, "Prabhjot Bharaj" wrote: > Hi, > > I am experiencing super slow throughput when using acks=-1 > Some further progress in continuation to the test in my previous

Re: Slow ISR catch-up

it is taking way longer. Where am I doing wrong? Thanks, Prabhjot On Fri, Sep 4, 2015 at 1:45 AM, Gwen Shapira wrote: > Yes, this should work. Expect lower throughput though. > > On Thu, Sep 3, 2015 at 12:52 PM, Prabhjot Bharaj > wrote: > > > Hi, > > > > Can I u

Re: Slow ISR catch-up

7;t wait for acks, so it > is possible that the messages were never acked... > > On Thu, Sep 3, 2015 at 7:56 AM, Prabhjot Bharaj > wrote: > > > Hi Folks, > > > > Request your expertise on my doubt here. > > > > *My setup:-* > > > > 5 node kaf

Slow ISR catch-up

Hi Folks, Request your expertise on my doubt here. *My setup:-* 5 node kafka cluster (4 cores, 8GB RAM) on RAID-6 (500 GB) Using Kafka 0.8.2.1 with modified ProducerPerformance.scala I've modified ProducerPerformance.scala to send custom ASCII data, instead of Byte Array of Zeroes *server.prope

Re: kafka producer-perf-test.sh compression-codec not working

ucer > produces, and the consumer looks at the difference between between the > timestamp in the body and the current timestamp. > > Or, if you were looking for ack latency, you can use the producer’s async > callback to measure latency. > -Erik > > From: Prabhjot Bharaj m

Re: kafka producer-perf-test.sh compression-codec not working

compression is concerned. If you do implement an > improvement, a patch might be helpful to the community. But something to > consider is that threwput alone isn’t the only important performance > measure. Round trip latency is also important. > Thanks, > -Erik > > > Fr

Re: Metrics to monitor in Kafka

This document explains some important metrics http://docs.confluent.io/1.0/kafka/monitoring.html Also, you can check the kafka demo at https://apps.sematext.com/demo. As a side thing, as I have done, you can report every stat to Ganglia and then see, according to your domain, what are the most i

Re: kafka producer-perf-test.sh compression-codec not working

> goal is for each message to be unique, but still have similar chunks that > a compression algorithm can detect and compress. > > -Erik > > > On 8/25/15, 6:47 AM, "Prabhjot Bharaj" wrote: > > >Hi, > > > >I have bene trying to use kafka-producer-

kafka producer-perf-test.sh compression-codec not working

Hi, I have bene trying to use kafka-producer-perf-test.sh to arrive at certain benchmarks. When I try to run it with --compression-codec values of 1, 2 and 3, I notice increased throughput compared to NoCompressionCodec But, When I checked the Producerperformance.scala, I saw that the the `produc

Raid vs individual disks

Hi, I've gone through the details mentioned about Raid and individual disks in the ops section of the documentation But, I would like to know what performance boost we can get with individual disks. Is anybody using Kafka with multiple disks or all are raid into 1 big disk ? Regards, Prabcs

Re: SocketTimeoutException with kafka-producer-perf-test.sh

This link already had the answer I was searching for: http://grokbase.com/t/kafka/users/145vmz70cb/java-net-sockettimeoutexception-in-broker On Fri, Aug 21, 2015 at 3:03 PM, Prabhjot Bharaj wrote: > Hi, > > Never mind. I have solved this problem by referring to an earlier > que

kafka-producer-perf-test.sh - No visible difference between request-num-acks 1 and -1

Hi, I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along the bundle I have setup a 5 machine cluster and on the same 5 machines, I'm also running zookeeper as well I am trying to see what is the maximum produce throughput I can get on this 5 node cluster I have created only

Re: SocketTimeoutException with kafka-producer-perf-test.sh

Hi, Never mind. I have solved this problem by referring to an earlier question, where Neha had suggested to use a higher value for request-timeout-ms Regards, prabcs On Fri, Aug 21, 2015 at 12:22 PM, Prabhjot Bharaj wrote: > Hello Folks, > > I'm using Kafka 0.8.2.1 with the def

SocketTimeoutException with kafka-producer-perf-test.sh

Hello Folks, I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along the bundle I have setup a 5 machine cluster and on the same 5 machines, I'm also running zookeeper as well I am trying to see what is the maximum produce throughput I can get on this 5 node cluster I have cre

Zookeeper jmx monitoring for kafka

Hi, Zookeeper monitoring - I was going through the 'Monitoring' section in http://kafka.apache.org/coding-guide.html I have setup Ganglia to monitor all the stats inocming from kafka jmx port But, in addition, there are a bunch of stats that Zookeeper also exposes in its jmx port. They are docume

Zookeeper use cases with Kafka

Hello Folks, I wish to contribute to Kafka internals. And, one of the things which can help me do that is understanding how kafka uses zookeeper. I have some of these basic doubts:- 1. Is zookeeper primarily used for locking ? If yes, in what cases and what kind of nodes does it use - sequential/

Re: Inconsistency with Zookeeper

Have you set broker id in zookeeper data directory and some unique broker Id in server.properties ? Regards, Prabhjot On Aug 6, 2015 1:43 PM, "Hemanth Abbina" wrote: > Hi, > > I am running a Kafka POC with below details > > * 3 Node cluster (4 Core, 16 GB RAM each) running Kafka 0.8.2.1

Re: Best practices - Using kafka (with http server) as source-of-truth

riety of other solutions) in > front of REST proxies for load balancing, HA, SSL termination, etc. This is > yet another hop, so it might affect throughput and latency. > > -Ewen > > On Mon, Jul 27, 2015 at 6:55 AM, Prabhjot Bharaj > wrote: > > > Hi Folks, > > > &g

Re: Connection to zk shell on Kafka

You need the jline JAR file that ships with ZooKeeper. > > Chris > > IBM Tivoli Systems > Research Triangle Park, NC > (919) 224-2240 > Internet: barl...@us.ibm.com > > > > From: Prabhjot Bharaj > To: users@kafka.apache.org, u...@zookeeper.apache

Connection to zk shell on Kafka

Hi folks, */kafka/bin# ./zookeeper-shell.sh localhost:2182/* *Connecting to localhost:2182/* *Welcome to ZooKeeper!* *JLine support is disabled* *WATCHER::* *WatchedEvent state:SyncConnected type:None path:null* ** I'm running 5 node zookeeper cluster on 5-node kafka cluster (each kafk

KAfka Mirror Maker

Hi, I'm using Mirror Maker with a cluster of 3 nodes and cluster of 5 nodes. I would like to ask - is the number of nodes a restriction for Mirror Maker? Also, are there any other restrictions or properties that should be common across both the clusters so that they continue mirroring. I'm aski

Re: Number of kafka topics/partitions supported per cluster of n nodes

@Jiefu Gong, Are the results of your tests available publicly? Regards, Prabhjot On Tue, Jul 28, 2015 at 10:35 PM, Prabhjot Bharaj wrote: > I would be using the servers available at my place of work. I dont have > access to AWS servers. I would starting off with a small number of no

Re: Number of kafka topics/partitions supported per cluster of n nodes

I'd > like to see these! > > On Mon, Jul 27, 2015 at 10:51 PM, Prabhjot Bharaj > wrote: > > > Hi, > > > > I'm looking forward to a benchmark which can explain how many total > number > > of topics and partitions can be created in a cluster of n nod

Number of kafka topics/partitions supported per cluster of n nodes

Hi, I'm looking forward to a benchmark which can explain how many total number of topics and partitions can be created in a cluster of n nodes, given the message size varies between x and y bytes and how does it vary with varying heap sizes and how it affects the system performance. e.g. the resu

Re: multiple producer throughput

Hi, Have you tried with acks=1 and -1 as well? Please share the numbers and the message size Regards, Prabcs On Jul 27, 2015 10:24 PM, "Yuheng Du" wrote: > Hi, > > I am running 40 producers on 40 nodes cluster. The messages are sent to 6 > brokers in another cluster. The producers are running P

Best practices - Using kafka (with http server) as source-of-truth

Hi Folks, I would like to understand the best practices when using kafka as the source-of-truth, given the fact that I want to pump in data to Kafka using http methods. What are the current production configurations for such a use case:- 1. Kafka-http-client - is it scalable the way Nginx is ??

Custom Zookeeper install with kafka

Hi, I've read on the Kafka documentation page that the zookeeper version used is 3.3.4 However, at my work, I've noticed certain problems with v3.3.4 (and the problems are documented here: http://zookeeper.apache.org/releases.html The latest stable version of zookeeper is 3.4.6 and has a lot of

parametrized number of partitions for topics

Hello Folks, I would like to create custom partitions for various topics e.g. topic1 - 100 partitions, topic2 - 10 partitions I have using this github repository - https://github.com/edenhill/librdkafka I had posted a similar question: https://github.com/edenhill/librdkafka/issues/308, but I got