anyone can give some suggestion? or an explanation why kafka give a big latency for large payload.
Thanks, Nan On Thu, Mar 14, 2019 at 3:53 PM Xu, Nan <n...@baml.com.invalid> wrote: > Hi, > > We are using kafka to send messages and there is less than 1% of > message is very big, close to 30M. understanding kafka is not ideal for > sending big messages, because the large message rate is very low, we just > want let kafka do it anyway. But still want to get a reasonable latency. > > To test, I just setup up a topic test on a single broker local kafka, > with only 1 partition and 1 replica, using the following command > > ./kafka-producer-perf-test.sh --topic test --num-records 2000000 > --throughput 1 --record-size 30000000 --producer.config > ../config/producer.properties > > Producer.config > > #Max 40M message > max.request.size=40000000 > buffer.memory=40000000 > > #2M buffer > send.buffer.bytes=2000000 > > 6 records sent, 1.1 records/sec (31.00 MB/sec), 973.0 ms avg latency, > 1386.0 max latency. > 6 records sent, 1.0 records/sec (28.91 MB/sec), 787.2 ms avg latency, > 1313.0 max latency. > 5 records sent, 1.0 records/sec (27.92 MB/sec), 582.8 ms avg latency, > 643.0 max latency. > 6 records sent, 1.1 records/sec (30.16 MB/sec), 685.3 ms avg latency, > 1171.0 max latency. > 5 records sent, 1.0 records/sec (27.92 MB/sec), 629.4 ms avg latency, > 729.0 max latency. > 5 records sent, 1.0 records/sec (27.61 MB/sec), 635.6 ms avg latency, > 673.0 max latency. > 6 records sent, 1.1 records/sec (30.09 MB/sec), 736.2 ms avg latency, > 1255.0 max latency. > 5 records sent, 1.0 records/sec (27.62 MB/sec), 626.8 ms avg latency, > 685.0 max latency. > 5 records sent, 1.0 records/sec (28.38 MB/sec), 608.8 ms avg latency, > 685.0 max latency. > > > On the broker, I change the > > socket.send.buffer.bytes=2024000 > # The receive buffer (SO_RCVBUF) used by the socket server > socket.receive.buffer.bytes=2224000 > > and all others are default. > > I am a little surprised to see about 1 s max latency and average about 0.5 > s. my understanding is kafka is doing the memory mapping for log file and > let system flush it. all the write are sequential. So flush should be not > affected by message size that much. Batching and network will take longer, > but those are memory based and local machine. my ssd should be far better > than 0.5 second. where the time got consumed? any suggestion? > > Thanks, > Nan > > > > > > > > ---------------------------------------------------------------------- > This message, and any attachments, is for the intended recipient(s) only, > may contain information that is privileged, confidential and/or proprietary > and subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer. If you are not the > intended recipient, please delete this message. >
