Re: JMS pub/sub (using Camel) performance issue with MQ in XA transactions
The cache shouldn’t be enabled, as we are using XA (as per Camel’s recommendation). We have also tuned the polling interval. That being said, what hurts the I/O subsystem on the MQ server is the Subscribe operation itself (not the polling for messages). I am wondering is if there would be any way (inside or outside of Camel) to have the XA transaction “scoped” on the REQUEST_MSGS requests instead of the SUBSCRIBE operation? So that each polling doesn’t do a subscribe operation every time and instead stays attached to an already opened subscription? As a reminder, I am attaching below how the XA transaction and subscription operations are being done (according to a tcpdump I analyzed): MQ Client --[XA_START]--> MQ Server MQ Client <--[XA_START_REPLY]-- MQ Server MQ Client --[SPI (SUBSCRIBE)]--> MQ Server MQ Client <--[SPI REPLY]-- MQ Server MQ Client --[REQUEST_MSGS]--> MQ Server MQ Client --[REQUEST_MSGS]--> MQ Server MQ Client <--[NOTIFICATION]-- MQ Server MQ Client --[MQCLOSE]--> MQ Server MQ Client <--[MQCLOSE_REPLY]-- MQ Server MQ Client --[XA_END]--> MQ Server MQ Client <--[XA_END_REPLY]-- MQ Server MQ Client --[XA_COMMIT]--> MQ Server MQ Client <--[XA_COMMIT_REPLY]-- MQ Server On Wed, Sep 4, 2019 at 05:46 Claus Ibsen wrote: > camel-jms uses spring jms, so there are many users with this combo > also with IBM MQ and XA. > Look at the many options spring jms has to tune its polling, and also > cache levels you can tweak. > There may be some more idle options you can set to make it "sleep" > longer when there are no messages. > But you can't avoid that spring jms is designed to keep polling for > new messages also when the queue is empty. > > > > On Tue, Sep 3, 2019 at 7:04 PM Benoit Fortin > wrote: > > > > We have a java application doing JMS subscriptions that is using Camel as > > its JMS provider. > > > > The application is subscribing to a topic using XA. It consumes 1 message > > in the queue, and then closes the XA transaction (each message is part of > > an XA transaction). Then the application re-attaches itself to the topic > to > > start over the same process for each message. > > > > When there are no messages to process, the client waits for 45 seconds > > (which is the XA transaction timeout) before closing the request and the > XA > > transaction and starting a new iteration. > > > > I have analyzed how this process is actually being done using a tcpdump, > > and here is what I found: > > > > —— > > > > MQ Client --[XA_START]--> MQ Server > > > > MQ Client <--[XA_START_REPLY]-- MQ Server > > > > > > MQ Client --[SPI (SUBSCRIBE)]--> MQ Server > > > > MQ Client <--[SPI REPLY]-- MQ Server > > > > > > MQ Client --[REQUEST_MSGS]--> MQ Server > > > > MQ Client --[REQUEST_MSGS]--> MQ Server > > > > MQ Client <--[NOTIFICATION]-- MQ Server > > > > > > MQ Client --[MQCLOSE]--> MQ Server > > > > MQ Client <--[MQCLOSE_REPLY]-- MQ Server > > > > > > MQ Client --[XA_END]--> MQ Server > > > > MQ Client <--[XA_END_REPLY]-- MQ Server > > > > > > MQ Client --[XA_COMMIT]--> MQ Server > > > > MQ Client <--[XA_COMMIT_REPLY]-- MQ Server > > > > —— > > > > This is process is on a durable subscription, so each of the iterations > > (even when there is no message to consume) ends up generating I/O on the > MQ > > server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, > where > > MQ keeps trace of its subscribers). > > > > Having a somewhat high number of subscribers doing this process, the MQ > > server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even > > outside of business hours, when there are no messages to consume). This > > process also consumes ~3 CPU outside of business hours. > > > > I am a bit puzzled that using XA on pub/sub scales so bad, and I am > > wondering if there is any way to implement this solution without doing so > > much subscribe operations. > > > > -- > Claus Ibsen > - > http://davsclaus.com @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 >
Re: JMS pub/sub (using Camel) performance issue with MQ in XA transactions
camel-jms uses spring jms, so there are many users with this combo also with IBM MQ and XA. Look at the many options spring jms has to tune its polling, and also cache levels you can tweak. There may be some more idle options you can set to make it "sleep" longer when there are no messages. But you can't avoid that spring jms is designed to keep polling for new messages also when the queue is empty. On Tue, Sep 3, 2019 at 7:04 PM Benoit Fortin wrote: > > We have a java application doing JMS subscriptions that is using Camel as > its JMS provider. > > The application is subscribing to a topic using XA. It consumes 1 message > in the queue, and then closes the XA transaction (each message is part of > an XA transaction). Then the application re-attaches itself to the topic to > start over the same process for each message. > > When there are no messages to process, the client waits for 45 seconds > (which is the XA transaction timeout) before closing the request and the XA > transaction and starting a new iteration. > > I have analyzed how this process is actually being done using a tcpdump, > and here is what I found: > > —— > > MQ Client --[XA_START]--> MQ Server > > MQ Client <--[XA_START_REPLY]-- MQ Server > > > MQ Client --[SPI (SUBSCRIBE)]--> MQ Server > > MQ Client <--[SPI REPLY]-- MQ Server > > > MQ Client --[REQUEST_MSGS]--> MQ Server > > MQ Client --[REQUEST_MSGS]--> MQ Server > > MQ Client <--[NOTIFICATION]-- MQ Server > > > MQ Client --[MQCLOSE]--> MQ Server > > MQ Client <--[MQCLOSE_REPLY]-- MQ Server > > > MQ Client --[XA_END]--> MQ Server > > MQ Client <--[XA_END_REPLY]-- MQ Server > > > MQ Client --[XA_COMMIT]--> MQ Server > > MQ Client <--[XA_COMMIT_REPLY]-- MQ Server > > —— > > This is process is on a durable subscription, so each of the iterations > (even when there is no message to consume) ends up generating I/O on the MQ > server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, where > MQ keeps trace of its subscribers). > > Having a somewhat high number of subscribers doing this process, the MQ > server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even > outside of business hours, when there are no messages to consume). This > process also consumes ~3 CPU outside of business hours. > > I am a bit puzzled that using XA on pub/sub scales so bad, and I am > wondering if there is any way to implement this solution without doing so > much subscribe operations. -- Claus Ibsen - http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
Re: JMS pub/sub (using Camel) performance issue with MQ in XA transactions
Hi, We do need XA as the client application is doing 2 phase commit in (1) the subscription and (2) a database. On Tue, Sep 3, 2019 at 14:49 Jean-Baptiste Onofré wrote: > Hi, > > What about using pure ACK mode ? > > You can see an example with queue (same can be applied to topic) here: > http://blog.nanthrax.net/?p=820 > > My point is: do you need XA as AFAIU you are using only JMS as resource > (XA is required when you want to use the same transaction with different > backends like JMS and database for instance) ? > > Regards > JB > > On 03/09/2019 19:03, Benoit Fortin wrote: > > We have a java application doing JMS subscriptions that is using Camel as > > its JMS provider. > > > > The application is subscribing to a topic using XA. It consumes 1 message > > in the queue, and then closes the XA transaction (each message is part of > > an XA transaction). Then the application re-attaches itself to the topic > to > > start over the same process for each message. > > > > When there are no messages to process, the client waits for 45 seconds > > (which is the XA transaction timeout) before closing the request and the > XA > > transaction and starting a new iteration. > > > > I have analyzed how this process is actually being done using a tcpdump, > > and here is what I found: > > > > —— > > > > MQ Client --[XA_START]--> MQ Server > > > > MQ Client <--[XA_START_REPLY]-- MQ Server > > > > > > MQ Client --[SPI (SUBSCRIBE)]--> MQ Server > > > > MQ Client <--[SPI REPLY]-- MQ Server > > > > > > MQ Client --[REQUEST_MSGS]--> MQ Server > > > > MQ Client --[REQUEST_MSGS]--> MQ Server > > > > MQ Client <--[NOTIFICATION]-- MQ Server > > > > > > MQ Client --[MQCLOSE]--> MQ Server > > > > MQ Client <--[MQCLOSE_REPLY]-- MQ Server > > > > > > MQ Client --[XA_END]--> MQ Server > > > > MQ Client <--[XA_END_REPLY]-- MQ Server > > > > > > MQ Client --[XA_COMMIT]--> MQ Server > > > > MQ Client <--[XA_COMMIT_REPLY]-- MQ Server > > > > —— > > > > This is process is on a durable subscription, so each of the iterations > > (even when there is no message to consume) ends up generating I/O on the > MQ > > server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, > where > > MQ keeps trace of its subscribers). > > > > Having a somewhat high number of subscribers doing this process, the MQ > > server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even > > outside of business hours, when there are no messages to consume). This > > process also consumes ~3 CPU outside of business hours. > > > > I am a bit puzzled that using XA on pub/sub scales so bad, and I am > > wondering if there is any way to implement this solution without doing so > > much subscribe operations. > > > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com >
Re: JMS pub/sub (using Camel) performance issue with MQ in XA transactions
Hi, What about using pure ACK mode ? You can see an example with queue (same can be applied to topic) here: http://blog.nanthrax.net/?p=820 My point is: do you need XA as AFAIU you are using only JMS as resource (XA is required when you want to use the same transaction with different backends like JMS and database for instance) ? Regards JB On 03/09/2019 19:03, Benoit Fortin wrote: > We have a java application doing JMS subscriptions that is using Camel as > its JMS provider. > > The application is subscribing to a topic using XA. It consumes 1 message > in the queue, and then closes the XA transaction (each message is part of > an XA transaction). Then the application re-attaches itself to the topic to > start over the same process for each message. > > When there are no messages to process, the client waits for 45 seconds > (which is the XA transaction timeout) before closing the request and the XA > transaction and starting a new iteration. > > I have analyzed how this process is actually being done using a tcpdump, > and here is what I found: > > —— > > MQ Client --[XA_START]--> MQ Server > > MQ Client <--[XA_START_REPLY]-- MQ Server > > > MQ Client --[SPI (SUBSCRIBE)]--> MQ Server > > MQ Client <--[SPI REPLY]-- MQ Server > > > MQ Client --[REQUEST_MSGS]--> MQ Server > > MQ Client --[REQUEST_MSGS]--> MQ Server > > MQ Client <--[NOTIFICATION]-- MQ Server > > > MQ Client --[MQCLOSE]--> MQ Server > > MQ Client <--[MQCLOSE_REPLY]-- MQ Server > > > MQ Client --[XA_END]--> MQ Server > > MQ Client <--[XA_END_REPLY]-- MQ Server > > > MQ Client --[XA_COMMIT]--> MQ Server > > MQ Client <--[XA_COMMIT_REPLY]-- MQ Server > > —— > > This is process is on a durable subscription, so each of the iterations > (even when there is no message to consume) ends up generating I/O on the MQ > server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, where > MQ keeps trace of its subscribers). > > Having a somewhat high number of subscribers doing this process, the MQ > server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even > outside of business hours, when there are no messages to consume). This > process also consumes ~3 CPU outside of business hours. > > I am a bit puzzled that using XA on pub/sub scales so bad, and I am > wondering if there is any way to implement this solution without doing so > much subscribe operations. > -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com
JMS pub/sub (using Camel) performance issue with MQ in XA transactions
We have a java application doing JMS subscriptions that is using Camel as its JMS provider. The application is subscribing to a topic using XA. It consumes 1 message in the queue, and then closes the XA transaction (each message is part of an XA transaction). Then the application re-attaches itself to the topic to start over the same process for each message. When there are no messages to process, the client waits for 45 seconds (which is the XA transaction timeout) before closing the request and the XA transaction and starting a new iteration. I have analyzed how this process is actually being done using a tcpdump, and here is what I found: —— MQ Client --[XA_START]--> MQ Server MQ Client <--[XA_START_REPLY]-- MQ Server MQ Client --[SPI (SUBSCRIBE)]--> MQ Server MQ Client <--[SPI REPLY]-- MQ Server MQ Client --[REQUEST_MSGS]--> MQ Server MQ Client --[REQUEST_MSGS]--> MQ Server MQ Client <--[NOTIFICATION]-- MQ Server MQ Client --[MQCLOSE]--> MQ Server MQ Client <--[MQCLOSE_REPLY]-- MQ Server MQ Client --[XA_END]--> MQ Server MQ Client <--[XA_END_REPLY]-- MQ Server MQ Client --[XA_COMMIT]--> MQ Server MQ Client <--[XA_COMMIT_REPLY]-- MQ Server —— This is process is on a durable subscription, so each of the iterations (even when there is no message to consume) ends up generating I/O on the MQ server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, where MQ keeps trace of its subscribers). Having a somewhat high number of subscribers doing this process, the MQ server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even outside of business hours, when there are no messages to consume). This process also consumes ~3 CPU outside of business hours. I am a bit puzzled that using XA on pub/sub scales so bad, and I am wondering if there is any way to implement this solution without doing so much subscribe operations.
Re: Camel performance issue
Thanks for replying. Here is the process: We copy 1000 xml files from source folder to destination folder in every test, for example from "c:/source" to "c:/dest“. Our camel route will consume those xml files, and convert them to JTO objects, at last send to JMS queue(ActiveMQ). Test 1 on physical machine, it takes more than 2 minutes. Test 2 on the VM which is hosted on the physical machine above, it takes about 30 seconds. We have tested this on several machines, always VM is faster. We are testing the performance of our product which is integrated with Camel, so we did a lot of test on different platforms. We are confused about performance gap between physical machine and VM. From the hawtio profiling we found that IO maybe the bottle neck, VM consumes files is faster that phycial machine. Our machines are powerful enough, so hardware should not be the problem I think. -- View this message in context: http://camel.465427.n5.nabble.com/Camel-performance-issue-tp5794918p5794957.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel performance issue
Well I have used the file component both in a VM, docker and a physical host and did not see any drop in performance. There must be something else on your physical host that is cause this. Have you done any profiling or enabled AuditEventNotifier on Camel to see which steps are taking the most time? Where is your ActiveMQ located? On Mon, Mar 6, 2017 at 2:56 PM, shuston [via Camel] < ml-node+s465427n5794958...@n5.nabble.com> wrote: > Did you run it multiple times in succession for each scenario? The file > system cache may skew your timing if you just run it once. > > > -Original Message- > > From: RichardChen [mailto:[hidden email] > <http:///user/SendEmail.jtp?type=node&node=5794958&i=0>] > > Sent: Monday, March 06, 2017 5:13 AM > > To: [hidden email] > <http:///user/SendEmail.jtp?type=node&node=5794958&i=1> > > Subject: Camel performance issue > > > > We are facing performance problem when we integrate camel into our > > product. > > The simple scenario is this: let camel monitor a folder, put 1000 xml > files to > > that folder, camel consume all the files and send them to ActiveMQ. > > > > I ran this scenario in both physical machine and VM(hosted on the same > > physical machine). The result is really confused me, the VM's > performance is > > higher than physical machine. > > I found that, camel on VM consumed files very fast, but meanwhie > physical > > machine was slow. > > > > Anthoer thing is our product pervious using Mule ESB, the performance is > > higher than camel when running the same scenario. > > > > Our testing evniroment is clean, no anti-virus software installed. Do OS > > version impact the result? > > We are using Windows7 x64 and Windows Server 2012 x64, camel is running > > on Apache Tomcat 8.5.6. > > > > Have you met the same problem when using camel? Or is there some > > advices on improving performance of my scenario? > > > > Thanks > > > > > > > > -- > > View this message in context: http://camel.465427.n5.nabble.com/Camel- > > performance-issue-tp5794918.html > > Sent from the Camel - Users mailing list archive at Nabble.com. > > > -- > If you reply to this email, your message will be added to the discussion > below: > http://camel.465427.n5.nabble.com/Camel-performance-issue- > tp5794918p5794958.html > To start a new topic under Camel - Users, email > ml-node+s465427n465428...@n5.nabble.com > To unsubscribe from Camel - Users, click here > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=c291Y2lhbmNlLmVxZGFtLnJhc2h0aUBnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=> > . > NAML > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://camel.465427.n5.nabble.com/Camel-performance-issue-tp5794918p5794960.html Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Camel performance issue
Did you run it multiple times in succession for each scenario? The file system cache may skew your timing if you just run it once. > -Original Message- > From: RichardChen [mailto:richard.c...@spsoft-cn.com] > Sent: Monday, March 06, 2017 5:13 AM > To: users@camel.apache.org > Subject: Camel performance issue > > We are facing performance problem when we integrate camel into our > product. > The simple scenario is this: let camel monitor a folder, put 1000 xml files to > that folder, camel consume all the files and send them to ActiveMQ. > > I ran this scenario in both physical machine and VM(hosted on the same > physical machine). The result is really confused me, the VM's performance is > higher than physical machine. > I found that, camel on VM consumed files very fast, but meanwhie physical > machine was slow. > > Anthoer thing is our product pervious using Mule ESB, the performance is > higher than camel when running the same scenario. > > Our testing evniroment is clean, no anti-virus software installed. Do OS > version impact the result? > We are using Windows7 x64 and Windows Server 2012 x64, camel is running > on Apache Tomcat 8.5.6. > > Have you met the same problem when using camel? Or is there some > advices on improving performance of my scenario? > > Thanks > > > > -- > View this message in context: http://camel.465427.n5.nabble.com/Camel- > performance-issue-tp5794918.html > Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel performance issue
You will need to be more specific about the performance. What was slow? How long did things take and what was your expectation? How are you putting the files? All at once? Please provide more metrics. Den 6 mars 2017 11:13 fm skrev "RichardChen [via Camel]" < ml-node+s465427n5794918...@n5.nabble.com>: We are facing performance problem when we integrate camel into our product. The simple scenario is this: let camel monitor a folder, put 1000 xml files to that folder, camel consume all the files and send them to ActiveMQ. I ran this scenario in both physical machine and VM(hosted on the same physical machine). The result is really confused me, the VM's performance is higher than physical machine. I found that, camel on VM consumed files very fast, but meanwhie physical machine was slow. Anthoer thing is our product pervious using Mule ESB, the performance is higher than camel when running the same scenario. Our testing evniroment is clean, no anti-virus software installed. Do OS version impact the result? We are using Windows7 x64 and Windows Server 2012 x64, camel is running on Apache Tomcat 8.5.6. Have you met the same problem when using camel? Or is there some advices on improving performance of my scenario? Thanks -- If you reply to this email, your message will be added to the discussion below: http://camel.465427.n5.nabble.com/Camel-performance-issue-tp5794918.html To start a new topic under Camel - Users, email ml-node+s465427n465428...@n5.nabble.com To unsubscribe from Camel - Users, click here <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=c291Y2lhbmNlLmVxZGFtLnJhc2h0aUBnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=> . NAML <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> -- View this message in context: http://camel.465427.n5.nabble.com/Camel-performance-issue-tp5794918p5794921.html Sent from the Camel - Users mailing list archive at Nabble.com.
Camel performance issue
We are facing performance problem when we integrate camel into our product. The simple scenario is this: let camel monitor a folder, put 1000 xml files to that folder, camel consume all the files and send them to ActiveMQ. I ran this scenario in both physical machine and VM(hosted on the same physical machine). The result is really confused me, the VM's performance is higher than physical machine. I found that, camel on VM consumed files very fast, but meanwhie physical machine was slow. Anthoer thing is our product pervious using Mule ESB, the performance is higher than camel when running the same scenario. Our testing evniroment is clean, no anti-virus software installed. Do OS version impact the result? We are using Windows7 x64 and Windows Server 2012 x64, camel is running on Apache Tomcat 8.5.6. Have you met the same problem when using camel? Or is there some advices on improving performance of my scenario? Thanks -- View this message in context: http://camel.465427.n5.nabble.com/Camel-performance-issue-tp5794918.html Sent from the Camel - Users mailing list archive at Nabble.com.
