Well, its possible that it is a networking issue which is why I asked for results from a test from a machine that isn¹t on the kafka cluster. And were all the producers run on the same member of the kafka cluster, or was it spread out?
Also, for the topic you are testing with, how many partitions does it have? And what is the replication factor? Also, how many kafka nodes do you have? On 9/29/15, 8:17 AM, "Prabhjot Bharaj" <[email protected]> wrote: >Hi Erik, > >I've not tested it on a producer that is not a part of the kafka cluster. > > >*Producer configuration on all machines:-* > >[email protected]:/a/kafka/config# cat producer.properties | egrep -v '^#|^$' > >metadata.broker.list=localhost:9092 > >producer.type=sync > >compression.codec=none > >serializer.class=kafka.serializer.DefaultEncoder > > >*Server configuration on all machines:-* > >[email protected]:/a/kafka/config# cat server.properties | egrep -v '^#|^$' > >broker.id=0 > >port=9092 > >num.network.threads=6 > >num.io.threads=8 > >socket.send.buffer.bytes=10485760 > >socket.receive.buffer.bytes=10485760 > >socket.request.max.bytes=104857600 > >log.dirs=/tmp/kafka-logs > >num.partitions=1 > >num.recovery.threads.per.data.dir=1 > >log.retention.hours=168 > >log.segment.bytes=1073741824 > >log.retention.check.interval.ms=300000 > >log.cleaner.enable=false > >zookeeper.connect=localhost:2181 > >zookeeper.connection.timeout.ms=6000 > >num.replica.fetchers=4 > >*Command used from both the machines (slow and fast producer):* > >kafka-producer-perf-test.sh --broker-list >x.x.x.x:9092,y.y.y.y:9092,z.z.z.z:9092,a.a.a.a:9092,b.b.b.b:9092 >--messages >1048576 --message-size 500 --topics part_1_repl_3_4 --show-detailed-stats >--threads 32 --request-num-acks 1 --batch-size 1000 --request-timeout-ms >10000 --compression-codec 2 --reporting-interval 1000 > >Regards, > >Prabhjot > >On Thu, Sep 24, 2015 at 6:36 PM, Helleren, Erik ><[email protected]> >wrote: > >> What happens when the new producer that is getting 70 MB/s is started >>on a >> machine that is not part of the kafka cluster? >> >> Can you include your topic description/configuration, producer >> configuration, and broker configuration? >> >> On 9/24/15, 1:44 AM, "Prabhjot Bharaj" <[email protected]> wrote: >> >> >Hi, >> > >> >I would like to dig deep into this issue. I've changed log4j.properties >> >for >> >logging in ALL mode in all places. I am getting lost in the logs. >> > >> >Any pointers would be welcome >> > >> >Please let me know if you would need any information regarding this >> > >> >Thanks, >> >Prabhjot >> > >> >On Wed, Sep 23, 2015 at 6:46 PM, Prabhjot Bharaj >><[email protected]> >> >wrote: >> > >> >> Hello Folks, >> >> >> >> I've noticed that 2 producer machines, that I had configured, have >> >>become >> >> very slow over time >> >> They are giving 17-19 MB/s >> >> >> >> But, a producer that I setup today is giving 70MB/s as the write >> >>throughput >> >> >> >> If I see the contents of bin, libs, config directories, nothing is >> >> different in the files on any of the producer machines. >> >> >> >> Producer is running on the kafka machines itself >> >> >> >> Request your expertise >> >> >> >> Regards, >> >> Prabhjot >> >> >> >> >> >> >> > >> > >> >-- >> >--------------------------------------------------------- >> >"There are only 10 types of people in the world: Those who understand >> >binary, and those who don't" >> >> > > >-- >--------------------------------------------------------- >"There are only 10 types of people in the world: Those who understand >binary, and those who don't"
