Hi Roshan, Just to narrow it down .. can u get rid of the bolt and see what numbers you get ? > Thanks, I will try that.
Which Kafka spout are you using (new/old) ? > The new one, from kafka-clients. I assume you mean msgs when you say ‘transactions’. > Yes that would be messages Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per worker ? > That would be the throughput of the whole topology, not per worker. Thanks, Bernard On Mon, Jan 7, 2019 at 4:24 PM Roshan Naik <roshan_n...@yahoo.com> wrote: > Just to narrow it down .. can u get rid of the bolt and see what numbers > you get ? > > Which Kafka spout are you using (new/old) ? > > I assume you mean msgs when you say ‘transactions’. > Did you mean you are getting 48k aggregate Msgs/sec or is that 48k per > worker ? > > > > Sent from Yahoo Mail for iPhone > <https://overview.mail.yahoo.com/?.src=iOS> > > On Sunday, January 6, 2019, 11:29 PM, Bernard Quizon < > bernard.qui...@stellarloyalty.com> wrote: > > Hi, > > Before anything else, I'm using Apache Storm 1.2.1 and Apache Kafka 1.1.1. > On my initial tests, I have: > > - *6 supervisor nodes * > - 1 worker per node > - 7 KafkaSpout executors per worker > - 1 bolt (that does nothing) per worker > - 0 Ackers > - 2 Kafka brokers with *42 partitions *on the topic > > With that configuration, no matter how I change some other configs (Please > see list below), my topology is capping out at around 48k TPS. > Please note that CPU usage, for the Supervisor nodes, is only around 20% > and Network usage is only around 20Mbps for both the Kafka Brokers and > Supervisor nodes, well below the network capacities. > > Now, I have increased the supervisor nodes from 6 to 12 and used a new > topic with 82 partitions, thinking that scaling out could help the > performance. > So is this the new configuration: > > - *12 supervisor nodes * > - 1 worker per node > - 7 KafkaSpout executors per worker > - 1 bolt (that does nothing) per worker > - 0 Ackers > - 2 Kafka brokers with *84 partitions* on the topic > > And I'm still getting around 48k TPS. > > Some other configs I played around with: > > - max.poll.records > - max.spout.pending > - processing.guarantee > - offset.commit.period.ms > - max.uncommitted.offsets > - poll.timeout.ms > - fetch.min.bytes > - fetch.max.bytes > - max.partition.fetch.bytes > - receive.buffer.bytes > - fetch.max.wait.ms > - topology.executor.receive.buffer.size > - topology.executor.send.buffer.size > - topology.receiver.buffer.size > - topology.transfer.buffer.size > > > Am I missing something here? Why is the throughput not improving? > > Just to add, I have also done some performance isolation tests on both > Kafka and Storm. > On a distributed consumer using Spark and Kafka, I was able to get around > 700K TPS (So we know that Kafka isn't the issue here). > And also I could get around 400k TPS on a custom Storm topology with 1 > spout that generates random transactions and 1 bolt that does nothing. > I feel like the numbers don't add up and the topology shouldn't be capping > out at around 48K TPS. > Your suggestions would be very much appreciated. > > > Thanks, > Bernard > >
