Hello Ted, Gordon,
When I say the JMS producers are sending synchronously, I mean they don't set
any options to the connection URL such as jms.forceAsyncSend. So I guess this
means the producer will wait for the settlement before sending message X + 1.
When I say it fails, I mean that with 1 producer, I have 2500 msg/s. When I add
a second producer, I am at 4800 msg/s (Which is roughly twice the throughput of
a single producer). But when I add a 3rd producer, I am at 5100 msg/s while I
except it to be around 7500 msg/s. So for me the scaling stops working when
adding a 3rd producer and above.
What you both explained to me about the single connection is indeed a plausible
candidate because in the tests of "broker only", the throughput of a single
connection is around 3 500 msg/s. So on a single connection, I shouldn't go
above that figure which is what I am seeing. I imagine that when I add more
producers/consumers, the throughput will shrink even more because the same
connection is used by all the producers and the consumers.
Do you think it might be an a good idea if the connections were per
workerThread and not only a single connection?
Another solution would be to use a maximum of 3 clients (producer or consumer)
per dispatcher and have a network of interconnected dispatchers but I find it
very heavy and hard to maintain and support on the client-side. Do you agree?
JMS Producer code
ConnectionFactory connectionFactory = new
JmsConnectionFactory("amqp://machine:port");
Connection connection = connectionFactory.createConnection();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Topic topic = session.createTopic("perf.topic");
messageProducer = session.createProducer(topic);
messageProducer.send(message);
Regards,
Adel
> Subject: Re: [Performance] Benchmarking Qpid dispatch router 0.6.0 with Qpid
> Java Broker 6.0.0
> To: users@qpid.apache.org
> From: tr...@redhat.com
> Date: Tue, 2 Aug 2016 13:42:24 -0400
>
>
>
> On 07/29/2016 08:40 AM, Adel Boutros wrote:
> > Hello Ted,
> >
> > Increasing the link capacity had no impact. So, I have
> > done a series of tests to try and isolate the issue.
> > We tested 3 different architecture without any consumers:
> > Producer --> Broker
> > Producer --> Dispatcher
> > Producer --> Dispatcher --> Broker
> > In every test, we sent 100 000 messages which contained a byte array of 100
> > bytes. The producers are sending in synchronous mode and with
> > AUTO_ACKNOWLEDGE.
> >
> > Our benchmark machines have 20 cores and 396 Gb Ram each. We have
> > currently put consumers/producers on 1 machine and dispatcher/brokers on
> > another machine. They are both connected with a 10 Gbps ethernet
> > connection. Nothing else is using the machines.
> >
> > The results are in
> > the table below.
> >
> > What I could observe:
> > The broker alone scales well when I add producers
> > The dispatcher alone scales well when I add producersThe dispatcher
> > connected to a broker scales well with 2 producersThe dispatcher connected
> > to a broker fails when having 3 producers or more
>
> In what way does it fail?
>
> >
> > I
> > also did some "qdstat -l" while the test was running and at max had 5
> > unsettled deliveries. So I don't think the problem comes with the
> > linkCapacity.
>
> You mentioned that you are running in synchronous mode. Does this mean
> that each producer is waiting for settlement on message X before sending
> message X+1?
>
> >
> > What else can we look at? How does the dispatcher connect the producers to
> > the broker? Does it open a new connection with each new producer? Or does
> > it use some sort of a connection pool?
>
> The router multiplexes the broker traffic over a single connection to
> the broker.
>
> >
> > Could the issue come from the capacity configuration of the link in the
> > connection between the broker and the dispatcher?
>
> Probably not in your case since the backlogs are much smaller than the
> default capacity.
>
> >
> >
> >
> >
> >
> >
> >
> >
> > Number of Producers
> > Broker
> > Dispatcher
> > Combined Producer Throughput (msg/s)
> > Combined Producer Latency (micros)
> >
> >
> > 1
> > YES
> >
> > NO
> >
> > 3 500
> > 370
> >
> >
> > 4
> > YES
> > NO
> >
> > 9 200
> > 420
> >
> >
> > 1
> > NO
> > YES
> > 6 000
> > 180
> >
> >
> > 2
> > NO
> > YES
> > 12 000
> > 192
> >
> >
> > 3
> > NO
> > YES
> > 16 000
> > 201
> >
> >
> > 1
> > YES
> > YES
> > 2 500
> > 360
> >
> >
> > 2
> > YES
> > YES
> > 4 800
> > 400
> >
> >
> > 3
> > YES
> > YES
> > 5 200
> > 540
> >
> >
> > qdstat -l
> > bash$ qdstat -b dell445srv:10254 -l
> > Router Links
> > type dir conn id id peer class addr phs cap
> > undel unsettled deliveries admin oper
> >
> > =======================================================================================================================
> > endpoint in 19 46 mobile perfQueue 1 250
> > 0 0 0 enabled up
> > endpoint out 19 54 mobile perf.topic 0
> > 250 0 2 4994922 enabled up
> > endpoint in 27 57 mobile perf.topic 0
> > 250 0 1 1678835 enabled up
> > endpoint in 28 58 mobile perf.topic 0
> > 250 0 1 1677653 enabled up
> > endpoint in 29 59 mobile perf.topic 0
> > 250 0 0 1638434 enabled up
> > endpoint in 47 94 mobile $management 0 250
> > 0 0 1 enabled up
> > endpoint out 47 95 local temp.2u+DSi+26jT3hvZ 250 0
> > 0 0 enabled up
> >
> > Regards,
> > Adel
> >
> >> Subject: Re: [Performance] Benchmarking Qpid dispatch router 0.6.0 with
> >> Qpid Java Broker 6.0.0
> >> To: users@qpid.apache.org
> >> From: tr...@redhat.com
> >> Date: Tue, 26 Jul 2016 10:32:29 -0400
> >>
> >> Adel,
> >>
> >> That's a good question. I think it's highly dependent on your
> >> requirements and the environment. Here are some random thoughts:
> >>
> >> - There's a trade-off between memory use (message buffering) and
> >> throughput. If you have many clients sharing the message bus,
> >> smaller values of linkCapacity will protect the router memory. If
> >> you have relatively few clients wanting to go fast, a larger
> >> linkCapacity is appropriate.
> >> - If the underlying network has high latency (satellite links, long
> >> distances, etc.), larger values of linkCapacity will be needed to
> >> protect against stalling caused by delayed settlement.
> >> - The default of 250 is considered a reasonable compromise. I think a
> >> value around 10 is better for a shared bus, but 500-1000 might be
> >> better for throughput with few clients.
> >>
> >> -Ted
> >>
> >>
> >> On 07/26/2016 10:08 AM, Adel Boutros wrote:
> >>> Thanks Ted,
> >>>
> >>> I will try to change linkCapacity. However, I was wondering if there is a
> >>> way to "calculate an optimal value for linkCapacity". What factors can
> >>> impact this field?
> >>>
> >>> Regards,
> >>> Adel
> >>>
> >>>> Subject: Re: [Performance] Benchmarking Qpid dispatch router 0.6.0 with
> >>>> Qpid Java Broker 6.0.0
> >>>> To: users@qpid.apache.org
> >>>> From: tr...@redhat.com
> >>>> Date: Tue, 26 Jul 2016 09:44:43 -0400
> >>>>
> >>>> Adel,
> >>>>
> >>>> The number of workers should be related to the number of available
> >>>> processor cores, not the volume of work or number of connections. 4 is
> >>>> probably a good number for testing.
> >>>>
> >>>> I'm not sure what the default link credit is for the Java broker (it's
> >>>> 500 for the c++ broker) or the clients you're using.
> >>>>
> >>>> The metric you should adjust is the linkCapacity for the listener and
> >>>> route-container connector. LinkCapacity is the number of deliveries
> >>>> that can be in-flight (unsettled) on each link. Qpid Dispatch Router
> >>>> defaults linkCapacity to 250. Depending on the volumes in your test,
> >>>> this might account for the discrepancy. You should try increasing this
> >>>> value.
> >>>>
> >>>> Note that linkCapacity is used to set initial credit for your links.
> >>>>
> >>>> -Ted
> >>>>
> >>>> On 07/25/2016 12:10 PM, Adel Boutros wrote:
> >>>>> Hello,We are actually running some performance benchmarks in an
> >>>>> architecture consisting of a Java Broker connected to a Qpid dispatch
> >>>>> router. We also have 3 producers and 3 consumers in the test. The
> >>>>> producers send message to a topic which has a binding on a queue with a
> >>>>> filter and the consumers receives message from that queue.
> >>>>> We have noticed a significant loss of performance in this architecture
> >>>>> compared to an architecture composed of a simple Java Broker. The
> >>>>> throughput of the producers is down to half and there are a lot of
> >>>>> oscillations in the presence of the dispatcher.
> >>>>>
> >>>>> I have tried to double the number of workers on the dispatcher but it
> >>>>> had no impact.
> >>>>>
> >>>>> Can you please help us find the cause of this issue?
> >>>>>
> >>>>> Dispacth router config
> >>>>> router {
> >>>>> id: router.10454
> >>>>> mode: interior
> >>>>> worker-threads: 4
> >>>>> }
> >>>>>
> >>>>> listener {
> >>>>> host: 0.0.0.0
> >>>>> port: 10454
> >>>>> role: normal
> >>>>> saslMechanisms: ANONYMOUS
> >>>>> requireSsl: no
> >>>>> authenticatePeer: no
> >>>>> }
> >>>>>
> >>>>> Java Broker config
> >>>>> export QPID_JAVA_MEM="-Xmx16g -Xms2g"
> >>>>> 1 Topic + 1 Queue
> >>>>> 1 AMQP port without any authentication mechanism (ANONYMOUS)
> >>>>>
> >>>>> Qdmanage on Dispatcher
> >>>>> qdmanage -b amqp://localhost:10454 create --type=address
> >>>>> prefix=perfQueue waypoint=true name=perf.queue.addr
> >>>>> qdmanage -b amqp://localhost:10454 create --type=address
> >>>>> prefix=perf.topic waypoint=true name=perf.topic.addr
> >>>>> qdmanage -b amqp://localhost:10454 create --type=connector
> >>>>> role=route-container addr=localhost port=10455
> >>>>> name=localhost.broker.10455.connector
> >>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink
> >>>>> addr=perfQueue dir=in connection=localhost.broker.10455.connector
> >>>>> name=localhost.broker.10455.perfQueue.in
> >>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink
> >>>>> addr=perf.topic dir=out connection=localhost.broker.10455.connector
> >>>>> name=localhost.broker.10455.perf.topic.out
> >>>>>
> >>>>> Combined producer throughput
> >>>>> 1 Broker: http://hpics.li/a9d6efa
> >>>>> 1 Broker + 1 Dispatcher: http://hpics.li/189299b
> >>>>>
> >>>>> Regards,
> >>>>> Adel
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> >>>> For additional commands, e-mail: users-h...@qpid.apache.org
> >>>>
> >>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> >> For additional commands, e-mail: users-h...@qpid.apache.org
> >>
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
