Thanks, Neal. This is my first foray into Pulsar, so you are probably right.
On Wed, Apr 19, 2023 at 3:34 PM Neal Feierabend <ne...@vt.edu> wrote: > Hello Steve, > > I'm a relative novice with both Pulsar and Camel so I could be completely > wrong, but I wonder if this may have more to do with Pulsar than Camel. > What kind of hardware are you testing on? Most of the benchmarks I've seen > that talk about 1 million or more messages/second are usually clusters with > really fast hardware. This one from Streamnative ( > https://streamnative.io/blog/apache-pulsar-vs-apache-kafka-2022-benchmark) > for example, is using 3 servers with 24 cores and 192G of memory for the > Pulsar brokers plus separate machines for the clients. Have you tried just > using the native Java client to see what kind of performance you get with > the same Pulsar cluster? You might also look at using Pulsar Perf ( > https://pulsar.apache.org/docs/2.11.x/performance-pulsar-perf/) to do some > performance testing to understand what you can expect from Pulsar on the > hardware you're using. Hope that helps some! > > Neal Feierabend (he/him/his) > *IT Developer Team Lead* > *Virginia Tech Transportation Institute* > > > On Wed, Apr 19, 2023 at 2:51 PM Steve973 <steve...@gmail.com> wrote: > > > Thanks, Claus. I have also tried Kafka in my app for comparison > purposes, > > and it seems to be a little bit more performant. > > > > I saw something called Starlight for JMS that does JMS over Pulsar, and > > gets about a million messages per second. I'm considering writing a > camel > > component for that. Im surprised, though, that I'm not getting anywhere > > near a million messages per second (or even a million messages in ten > > seconds) with either Pulsar or Kafka. With a small payload, what > > throughput should I be seeing? > > > > On Wed, Apr 19, 2023, 4:40 AM Claus Ibsen <claus.ib...@gmail.com> wrote: > > > > > Hi > > > > > > I think camel-pulsar does an ACK per message at the end of routing. > > > Then the performance may become slower. > > > > > > kafka uses async commit every 5 sec in the background (by default). > > > > > > for camel-pulsar you can also use manual ack, and then find a way to > > batch > > > acks. > > > But maybe there is a way in pulsar to also ack like kafka via a > > background > > > task. > > > And maybe we can improve camel-pulsar to make this easier to do out of > > the > > > box. > > > > > > Of course if you do async acks, then you can have duplicates in case > the > > > app crashes and is restarted, > > > then pulsar will resume from last known "ack" position. > > > > > > > > > > > > On Mon, Apr 17, 2023 at 5:04 PM Steve973 <steve...@gmail.com> wrote: > > > > > > > Hello. I have been experimenting with the Camel Pulsar component as > a > > > more > > > > performant alternative to traditional JMS brokers. I have seen > > > performance > > > > comparisons that set Pulsar even above Kafka in most cases. It is > > > reported > > > > that Pulsar can handle (~3.5) millions of messages per second. In my > > use > > > > case, I am sending very simple messages, where I have a couple of > > > headers, > > > > and a payload that is a simple POJO with a string field and a map > with > > > > between one and five entries, depending on the message. I am using > > > > protobuf to de/serialize the message body. I am seeing approximately > > one > > > > thousand messages per second. I can only assume that it is "user > > error" > > > on > > > > my part, but I was wondering if any of you have an example that > > > > demonstrates performance that is more on-par with the advertised > > message > > > > rate. If not, how can I determine what is slowing down Pulsar's > > > > performance in my use case? > > > > > > > > Thanks, > > > > Steve > > > > > > > > > > > > > -- > > > Claus Ibsen > > > ----------------- > > > @davsclaus > > > Camel in Action 2: https://www.manning.com/ibsen2 > > > > > >
