Re: kafka local single partition, what's the limit?

2018-08-22 Thread Nan Xu
the data directory is memory basd, no hard drive involved. mount -t tmpfs -o size=25G tmpfs /mnt/ramdisk and use this as data folder. iostat show 0 write too. On Wed, Aug 22, 2018 at 11:06 PM Ken Chen wrote: > You are reaching 10gb * 1000 / 64 = 156 MB / s which probably saturated > your hard

Re: kafka local single partition, what's the limit?

2018-08-22 Thread Ken Chen
You are reaching 10gb * 1000 / 64 = 156 MB / s which probably saturated your hard drive bandwidth ? So you can take a look at your iostats -- Sent from my iPhone On Aug 22, 2018, at 8:20 PM, Nan Xu wrote: I setup a local single node test. producer and broker are sitting at the same VM.

kafka local single partition, what's the limit?

2018-08-22 Thread Nan Xu
I setup a local single node test. producer and broker are sitting at the same VM. broker only has a single node(localhost) and a single partition. producer produce message as fast as it could in a single thread. all update to a SINGLE key(String). the kafka broker data directory is memory based