Is your network shared? Is so, another possibility is that some other apps are consuming the bandwidth.
Thanks, Jun On Sun, Apr 21, 2013 at 12:23 PM, Andrew Neilson <arsneil...@gmail.com>wrote: > Thanks very much for the reply Neha! So I swapped out the consumer that > processes the messages with one that just prints them. It does indeed > achieve a much better rate at peaks but can still nearly zero out (if not > completely zero out). I plotted the messages printed in graphite to show > the behaviour I'm seeing (this is messages printed per second): > > > https://www.dropbox.com/s/7u7uyrefw6inetu/Screen%20Shot%202013-04-21%20at%2011.44.38%20AM.png > > The peaks are over ten thousand per second and the troughs can go below 10 > per second just prior to another peak. I know that there are plenty of > messages available because the ones currently being processed are still > from Friday afternoon, so this may or may not have something to do with > this pattern. > > Is there anything I can do to avoid the periods of lower performance? > Ideally I would be processing messages as soon as they are written. > > > On Sun, Apr 21, 2013 at 8:49 AM, Neha Narkhede <neha.narkh...@gmail.com > >wrote: > > > Some of the reasons a consumer is slow are - > > 1. Small fetch size > > 2. Expensive message processing > > > > Are you processing the received messages in the consumer ? Have you > > tried running console consumer for this topic and see how it performs > > ? > > > > Thanks, > > Neha > > > > On Sun, Apr 21, 2013 at 1:59 AM, Andrew Neilson <arsneil...@gmail.com> > > wrote: > > > I am currently running a deployment with 3 brokers, 3 ZK, 3 producers, > 2 > > > consumers, and 15 topics. I should first point out that this is my > first > > > project using Kafka ;). The issue I'm seeing is that the consumers are > > only > > > processing about 15 messages per second from what should be the largest > > > topic it is consuming (we're sending 200-400 ~300 byte messages per > > second > > > to this topic). I should note that I'm using a high level ZK consumer > and > > > ZK 3.4.3. > > > > > > I have a strong feeling I have not configured things properly so I > could > > > definitely use some guidance. Here is my broker configuration: > > > > > > brokerid=1 > > > port=9092 > > > socket.send.buffer=1048576 > > > socket.receive.buffer=1048576 > > > max.socket.request.bytes=104857600 > > > log.dir=/home/kafka/data > > > num.partitions=1 > > > log.flush.interval=10000 > > > log.default.flush.interval.ms=1000 > > > log.default.flush.scheduler.interval.ms=1000 > > > log.retention.hours=168 > > > log.file.size=536870912 > > > enable.zookeeper=true > > > zk.connect=XXX > > > zk.connectiontimeout.ms=1000000 > > > > > > Here is my producer config: > > > > > > zk.connect=XXX > > > producer.type=async > > > compression.codec=0 > > > > > > Here is my consumer config: > > > > > > zk.connect=XXX > > > zk.connectiontimeout.ms=100000 > > > groupid=XXX > > > autooffset.reset=smallest > > > socket.buffersize=1048576 > > > fetch.size=10485760 > > > queuedchunks.max=10000 > > > > > > Thanks for any assistance you can provide, > > > > > > Andrew > > >
