The cache shouldn’t be enabled, as we are using XA (as per Camel’s
recommendation). We have also tuned the polling interval.
That being said, what hurts the I/O subsystem on the MQ server is the
Subscribe operation itself (not the polling for messages).
I am wondering is if there would be any way (inside or outside of Camel) to
have the XA transaction “scoped” on the REQUEST_MSGS requests instead of
the SUBSCRIBE operation? So that each polling doesn’t do a subscribe
operation every time and instead stays attached to an already opened
subscription?
As a reminder, I am attaching below how the XA transaction and subscription
operations are being done (according to a tcpdump I analyzed):
MQ Client --[XA_START]--> MQ Server
MQ Client <--[XA_START_REPLY]-- MQ Server
MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
MQ Client <--[SPI REPLY]-- MQ Server
MQ Client --[REQUEST_MSGS]--> MQ Server
MQ Client --[REQUEST_MSGS]--> MQ Server
MQ Client <--[NOTIFICATION]-- MQ Server
MQ Client --[MQCLOSE]--> MQ Server
MQ Client <--[MQCLOSE_REPLY]-- MQ Server
MQ Client --[XA_END]--> MQ Server
MQ Client <--[XA_END_REPLY]-- MQ Server
MQ Client --[XA_COMMIT]--> MQ Server
MQ Client <--[XA_COMMIT_REPLY]-- MQ Server
On Wed, Sep 4, 2019 at 05:46 Claus Ibsen <claus.ib...@gmail.com> wrote:
> camel-jms uses spring jms, so there are many users with this combo
> also with IBM MQ and XA.
> Look at the many options spring jms has to tune its polling, and also
> cache levels you can tweak.
> There may be some more idle options you can set to make it "sleep"
> longer when there are no messages.
> But you can't avoid that spring jms is designed to keep polling for
> new messages also when the queue is empty.
>
>
>
> On Tue, Sep 3, 2019 at 7:04 PM Benoit Fortin <benoit.for...@gmail.com>
> wrote:
> >
> > We have a java application doing JMS subscriptions that is using Camel as
> > its JMS provider.
> >
> > The application is subscribing to a topic using XA. It consumes 1 message
> > in the queue, and then closes the XA transaction (each message is part of
> > an XA transaction). Then the application re-attaches itself to the topic
> to
> > start over the same process for each message.
> >
> > When there are no messages to process, the client waits for 45 seconds
> > (which is the XA transaction timeout) before closing the request and the
> XA
> > transaction and starting a new iteration.
> >
> > I have analyzed how this process is actually being done using a tcpdump,
> > and here is what I found:
> >
> > ——
> >
> > MQ Client --[XA_START]--> MQ Server
> >
> > MQ Client <--[XA_START_REPLY]-- MQ Server
> >
> >
> > MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
> >
> > MQ Client <--[SPI REPLY]-- MQ Server
> >
> >
> > MQ Client --[REQUEST_MSGS]--> MQ Server
> >
> > MQ Client --[REQUEST_MSGS]--> MQ Server
> >
> > MQ Client <--[NOTIFICATION]-- MQ Server
> >
> >
> > MQ Client --[MQCLOSE]--> MQ Server
> >
> > MQ Client <--[MQCLOSE_REPLY]-- MQ Server
> >
> >
> > MQ Client --[XA_END]--> MQ Server
> >
> > MQ Client <--[XA_END_REPLY]-- MQ Server
> >
> >
> > MQ Client --[XA_COMMIT]--> MQ Server
> >
> > MQ Client <--[XA_COMMIT_REPLY]-- MQ Server
> >
> > ——
> >
> > This is process is on a durable subscription, so each of the iterations
> > (even when there is no message to consume) ends up generating I/O on the
> MQ
> > server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue,
> where
> > MQ keeps trace of its subscribers).
> >
> > Having a somewhat high number of subscribers doing this process, the MQ
> > server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even
> > outside of business hours, when there are no messages to consume). This
> > process also consumes ~3 CPU outside of business hours.
> >
> > I am a bit puzzled that using XA on pub/sub scales so bad, and I am
> > wondering if there is any way to implement this solution without doing so
> > much subscribe operations.
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
