Re: what's the relationship between Zookeeper and Kafka ?

In addition to what Michael noted, this question has been asked a few times before too and here's one such previous discussion https://www.quora.com/What-is-the-actual-role-of-ZooKeeper-in-Kafka -Jaikiran On Wednesday 14 September 2016 03:50 AM, Michael Noll wrote: Eric, the latest versions

Re: what's the relationship between Zookeeper and Kafka ?

Eric, the latest versions of Kafka use ZooKeeper only on the side of the Kafka brokers, i.e. the servers in a Kafka cluster. Background: In older versions of Kafka, the Kafka consumer API required client applications (that would read from data Kafka) to also talk to ZK. Why would they need to

Re: what's the relationship between Zookeeper and Kafka ?

AFAIK Kafka uses Zookeeper to coordinate the Kafka clusters ( set of brokers ). Consumers usually connect Zookeeper to retrieve the list of brokers. Then connect the broker. *Valerio* On 10 September 2016 at 22:11, Eric Ho wrote: > I notice that some Spark programs

what's the relationship between Zookeeper and Kafka ?

I notice that some Spark programs would contact something like 'zoo1:2181' when trying to suck data out of Kafka. Does the kafka data actually get routed out of zookeeper before delivering the payload onto Spark ? -- -eric ho

Re: Relationship between Zookeeper and Kafka

As a test, why not just use a disk with provisioned IOPs of 4000? Just as a test - see if it improves. Also, you have not supplied any metrics regarding the VM's performance. Is the CPU busy? Is IO maxed out? Network? Disk? Use a tool like atop, and tell us what you find. Philip On May 20,

Re: Relationship between Zookeeper and Kafka

Philip, Thanks for the response. I used top yesterday and determined that part of my problem was that the kafaka shell script is pre-configured to only use 512M of RAM, and thus it wasn't using memory efficiently. That has helped out tremendously. Adding an echo at the start of the script that it

Re: Relationship between Zookeeper and Kafka

My guess, EBS is likely your bottleneck. Try running on instance local disks, and compare your results. Is this 0.8? What replication factor are you using? On Mon, May 20, 2013 at 8:11 AM, Jason Weiss jason_we...@rapid7.com wrote: I'm trying to maximize my throughput and seem to have hit a

Re: Relationship between Zookeeper and Kafka

Ahh, yeah, piops is definitely faster than standard EBS, but still much slower than local disk. you could try benchmarking local disk to see what the instances you are using are capable of, then try tweaking iops etc to see where you get. M1.Larges arent super fast so your macbook beating them

Re: Relationship between Zookeeper and Kafka

Hi Jason, On May 20, 2013, at 10:01am, Jason Weiss wrote: Hi Scott. I'm using Kafka 0.7.2. I am using the default replication factor, since I don't recall changing that configuration at all. I'm using provisioned IOPS, which from attending the AWS event in NYC a few weeks ago was