Hi Jiefu, Gwen,
I am running the Throughput versus stored data test:
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
test 50000000000 100 -1 acks=1 bootstrap.servers=
esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196
After around 50,000,000 messages were sent, I got a bunch of connection
refused error as I mentioned before. I checked the logs on the broker and
here is what I see:
[2015-07-14 15:11:23,578] WARN Partition [test,4] on broker 5: No
checkpointed highwatermark is found for partition [test,4]
(kafka.cluster.Partition)
[2015-07-14 15:12:33,298] INFO Rolled new log segment for 'test-4' in 4 ms.
(kafka.log.Log)
[2015-07-14 15:12:33,299] INFO Rolled new log segment for 'test-0' in 1 ms.
(kafka.log.Log)
[2015-07-14 15:13:39,529] INFO Rolled new log segment for 'test-4' in 1 ms.
(kafka.log.Log)
[2015-07-14 15:13:39,531] INFO Rolled new log segment for 'test-0' in 1 ms.
(kafka.log.Log)
[2015-07-14 15:14:48,502] INFO Rolled new log segment for 'test-4' in 3 ms.
(kafka.log.Log)
[2015-07-14 15:14:48,502] INFO Rolled new log segment for 'test-0' in 1 ms.
(kafka.log.Log)
[2015-07-14 15:15:51,478] INFO Rolled new log segment for 'test-4' in 1 ms.
(kafka.log.Log)
[2015-07-14 15:15:51,479] INFO Rolled new log segment for 'test-0' in 1 ms.
(kafka.log.Log)
[2015-07-14 15:16:52,589] INFO Rolled new log segment for 'test-4' in 1 ms.
(kafka.log.Log)
[2015-07-14 15:16:52,590] INFO Rolled new log segment for 'test-0' in 1 ms.
(kafka.log.Log)
[2015-07-14 15:17:57,406] INFO Rolled new log segment for 'test-4' in 1 ms.
(kafka.log.Log)
[2015-07-14 15:17:57,407] INFO Rolled new log segment for 'test-0' in 0 ms.
(kafka.log.Log)
[2015-07-14 15:18:39,792] FATAL [KafkaApi-5] Halting due to unrecoverable
I/O error while handling produce request: (kafka.server.KafkaApis)
kafka.common.KafkaStorageException: I/O exception in append to log 'test-0'
at kafka.log.Log.append(Log.scala:266)
at
kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:379)
at
kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:365)
at kafka.utils.Utils$.inLock(Utils.scala:535)
at kafka.utils.Utils$.inReadLock(Utils.scala:541)
at
kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:365)
at
kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:291)
at
kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:282)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at scala.coll
Can you help me with this problem? Thanks.
On Tue, Jul 14, 2015 at 5:12 PM, Yuheng Du <yuheng.du.h...@gmail.com> wrote:
> I checked the logs on the brokers, it seems that the zookeeper or the
> kafka server process is not running on this broker...Thank you guys. I will
> see if it happens again.
>
> On Tue, Jul 14, 2015 at 4:53 PM, JIEFU GONG <jg...@berkeley.edu> wrote:
>
>> Hmm..yeah some error logs would be nice like Gwen pointed out. Do any of
>> your brokers fall out of the ISR when sending messages? It seems like your
>> setup should be fine, so I'm not entirely sure.
>>
>> On Tue, Jul 14, 2015 at 1:31 PM, Yuheng Du <yuheng.du.h...@gmail.com>
>> wrote:
>>
>> > Jiefu,
>> >
>> > I am performing these tests on a 6 nodes cluster in cloudlab (a
>> > infrastructure built for scientific research). I use 2 nodes as
>> producers,
>> > 2 as brokers only, and 2 as consumers. I have tested for each individual
>> > machines and they work well. I did not use AWS. Thank you!
>> >
>> > On Tue, Jul 14, 2015 at 4:20 PM, JIEFU GONG <jg...@berkeley.edu> wrote:
>> >
>> > > Yuheng, are you performing these tests locally or using a service
>> such as
>> > > AWS? I'd try using each separate machine individually first,
>> connecting
>> > to
>> > > the ZK/Kafka servers and ensuring that each is able to first log and
>> > > consume messages independently.
>> > >
>> > > On Tue, Jul 14, 2015 at 1:17 PM, Gwen Shapira <gshap...@cloudera.com>
>> > > wrote:
>> > >
>> > > > Are there any errors on the broker logs?
>> > > >
>> > > > On Tue, Jul 14, 2015 at 11:57 AM, Yuheng Du <
>> yuheng.du.h...@gmail.com>
>> > > > wrote:
>> > > > > Jiefu,
>> > > > >
>> > > > > Thank you. The three producers can run at the same time. I mean
>> > should
>> > > > they
>> > > > > be started at exactly the same time? (I have three consoles from
>> each
>> > > of
>> > > > > the three machines and I just start the console command manually
>> one
>> > by
>> > > > > one) Or a small variation of the starting time won't matter?
>> > > > >
>> > > > > Gwen and Jiefu,
>> > > > >
>> > > > > I have started the three producers at three machines, after a
>> while,
>> > > all
>> > > > of
>> > > > > them gives a java.net.ConnectException:
>> > > > >
>> > > > > [2015-07-14 12:56:46,352] WARN Error in I/O with producer0-link-0/
>> > > > > 192.168.1.1 (org.apache.kafka.common.network.Selector)
>> > > > >
>> > > > > java.net.ConnectException: Connection refused......
>> > > > >
>> > > > > [2015-07-14 12:56:48,056] WARN Error in I/O with producer1-link-0/
>> > > > > 192.168.1.2 (org.apache.kafka.common.network.Selector)
>> > > > >
>> > > > > java.net.ConnectException: Connection refused.....
>> > > > >
>> > > > > What could be the cause?
>> > > > >
>> > > > > Thank you guys!
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Tue, Jul 14, 2015 at 2:47 PM, JIEFU GONG <jg...@berkeley.edu>
>> > > wrote:
>> > > > >
>> > > > >> Yuheng,
>> > > > >>
>> > > > >> Yes, if you read the blog post it specifies that he's using three
>> > > > separate
>> > > > >> machines. There's no reason the producers cannot be started at
>> the
>> > > same
>> > > > >> time, I believe.
>> > > > >>
>> > > > >> On Tue, Jul 14, 2015 at 11:42 AM, Yuheng Du <
>> > yuheng.du.h...@gmail.com
>> > > >
>> > > > >> wrote:
>> > > > >>
>> > > > >> > Hi,
>> > > > >> >
>> > > > >> > I am running the performance test for kafka.
>> > > > >> > https://gist.github.com/jkreps
>> > > > >> > /c7ddb4041ef62a900e6c
>> > > > >> >
>> > > > >> > For the "Three Producers, 3x async replication" scenario, the
>> > > command
>> > > > is
>> > > > >> > the same as one producer:
>> > > > >> >
>> > > > >> > bin/kafka-run-class.sh
>> > > > org.apache.kafka.clients.tools.ProducerPerformance
>> > > > >> > test 50000000 100 -1 acks=1
>> > > > >> > bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092
>> > > > >> > buffer.memory=67108864 batch.size=8196
>> > > > >> >
>> > > > >> > So How to I run the test for three producers? Do I just run
>> them
>> > on
>> > > > three
>> > > > >> > separate servers at the same time? Will there be some error in
>> > this
>> > > > way
>> > > > >> > since the three producers can't be started at the same time?
>> > > > >> >
>> > > > >> > Thanks.
>> > > > >> >
>> > > > >> > best,
>> > > > >> > Yuheng
>> > > > >> >
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >>
>> > > > >> Jiefu Gong
>> > > > >> University of California, Berkeley | Class of 2017
>> > > > >> B.A Computer Science | College of Letters and Sciences
>> > > > >>
>> > > > >> jg...@berkeley.edu <elise...@berkeley.edu> | (925) 400-3427
>> > > > >>
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Jiefu Gong
>> > > University of California, Berkeley | Class of 2017
>> > > B.A Computer Science | College of Letters and Sciences
>> > >
>> > > jg...@berkeley.edu <elise...@berkeley.edu> | (925) 400-3427
>> > >
>> >
>>
>>
>>
>> --
>>
>> Jiefu Gong
>> University of California, Berkeley | Class of 2017
>> B.A Computer Science | College of Letters and Sciences
>>
>> jg...@berkeley.edu <elise...@berkeley.edu> | (925) 400-3427
>>
>
>
