Can you send high priority messages down their own channels and the persistent ones down their own.
These would run as separate processes and (possibly) not block each other. Regards John. -----Original Message----- From: Potkay, Peter M (PLC, IT) [mailto:[EMAIL PROTECTED] Sent: 30 May 2003 14:21 To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi stent messages So here is my real question, which is what makes me wonder exactly how a QM handles messages. Our HUB server is using Veritas. The disk that is being written to (whenever that may be) is actually on the Storage Area Network (SAN). The HUB is also clustered with 2 queue managers dedicated to MQSI. The HUB acts simply as a gateway queue manager for this MQSI cluster. THE MQSI boxes are in 2 separate locations, with Veritas, and both again also write to the SAN. Whenever we make bin changes to the SAN, that change ripples across the fabric, making the SAN unavailable for a tiny bit of time. Now, we have an app that is counting milliseconds in its roundtrip of the message. This message starts on one of the spokes, comes to the HUB, is round robined to one of the MQSI boxes, the processed message comes back to the HUB, which then sends it down to the receiving spoke. It processes the message, sends it back to the hub, round robined into MQSI for processing the reply, the processed reply goes back to the hub, which then sends the reply back to the originating spoke. For 99.99% of the time, this roundtrip takes under 500 milliseconds. The app waits up to 2000 milliseconds for the reply. The messages are non persistent and about 25K in size. Whenever the bin changes to the SAN take place, we start getting messages that take longer than 2000 milliseconds, and now we have orphaned replies. These are non persistent messages that are under 64K, so why does a change that makes the disk unavailable cause these messages to slow down? My guess is that the persistent messages the HUB is processing at the same time (or the >64K Nonpersistent ones) must somehow be effecting the performance of the non persistent ones. And I also assume that channel speed has nothing to do with this. So the angle I am after here is how can I increase the performance of my messages for this app so that changes to the SAN don't effect it. -----Original Message----- From: John Scott [mailto:[EMAIL PROTECTED] Sent: Friday, May 30, 2003 10:18 AM To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi stent messages This would be true for the smallest unit of execution (normally a thread, not a process). Thus if a thread of execution was committing data to disk, that thread would not continue until the commit was completed. However, another thread within the same process would get CPU if it was able to execute some code. It all depends on whether the MQ processes are multi-threaded or not. Regards John. -----Original Message----- From: Potkay, Peter M (PLC, IT) [mailto:[EMAIL PROTECTED] Sent: 30 May 2003 12:50 To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi stent messages Gary, About a queue manager, or any process that runs on any computer, I have always assumed, maybe incorrectly, that a CPU on a box can only do one thing at a time. It may be incredibly fast, giving the illusion of many things happening at once, but when you get right down to it, everything has to wait its turn for the CPU to do its thing. And if the CPU is waiting to interact with the disk to write a persistent (or bigger than 64k non-persistent) message (I guess even if that message is going to blink in and out of a QM Alias or XMITQ???) then the CPU cant go on to handle non persistent messages. ??? -----Original Message----- From: Gary Ward [mailto:[EMAIL PROTECTED] Sent: Thursday, May 29, 2003 7:47 PM To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persistent messages Peter, I'll gladly defer to any IBM MQ Developers lurking on the list since this is getting really 'down and dirty', but in the meantime I'll give you another $.02... Right about my answer to Q2... I guess I missed your point. Having read your follow-up, I would offer this theory. I think the queue manager MUST be able to do more than a single task at once. That's why it's made up of many individual processes which have dedicated tasks. I would think at any one moment there's a bunch of messages floating around in various states. If the messages are non-persistent, they're not logged and hence can be available immediately (if they're not IN SYNCPOINT with persistent messages). Non-persistent messages only get written to disk when there's not enough memory available to hold them on an individual queue. I'm not sure about persistent messages always going right to the queue file system, but my gut feeling is that they do. If your non-persistent messages are 100K and you have your queues set at the 64KB default, I'm pretty sure they go right to disk. That's why you should tune that non-persistent message buffer. Hopefully something is already waiting for these non-persistent messages so they get handed off directly in the manner that T. Rob and I mentioned earlier in this thread. Then there's no I/O at all. Hope this helps... any IBMer's want to comment???? -----Original Message----- From: MQSeries List [mailto:[EMAIL PROTECTED] Behalf Of Potkay, Peter M (PLC, IT) Sent: Thursday, May 29, 2003 6:11 PM To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi st m essages Gary, I think you answer to Q2 pertains more to when a message is about to leave a server to go onto the next one. That setting tells the receiving side how soon it can have the Nonpersistent message in relation to the batch of messages coming across the channel. I am curious about once the messages have already been accepted on the HUB. Whether they were persistent or not, regardless of the channel speed setting, at any given moment on our busy HUB, the QM finds itself with lots and lots of messages that it now has complete control over. As it routes them thru the QMAliases and the XMITQs, it has to "stop" and log the persistent messages. I feel that this activity must somehow also effect the non persistent ones as well, since the QM can only do one thing at a time, regardless how fast it does it. If it is busy logging a persistent message, it can't route a non persistent one at that exact moment, correct? Regarding the queue buffer setting, if my messages are less than 64K, and because due to high activity all my channels in a particular SPOKE-HUB-SPOKE route are running, then a non persistent message would go in and out of each XMIT queue, in and out of each QM Alias queue and in and out of each application queue (assuming the app has an outstanding GET with wait) with no I/O to the disk? What if the messages are 100K non persistent ones and the buffer setting is still at default? Are you saying that a non persistent message is still written to disk? If yes, to me that sounds like there is no reason to not use persistence always on any message larger than 64K. Surely that can't be the case! Or is it like T.Rob suggested: Nonpersistent gets written to disk, persistent gets written to disk AND log, for a double I/O??? -----Original Message----- From: Gary Ward [mailto:[EMAIL PROTECTED] Sent: Thursday, May 29, 2003 1:19 PM To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi st m essages Let me throw in my $.02 - which is actually less in Euros lately ;) Q1. There "could" be I/O to the queue filesystem... see more below Q2. The non-persistent messages will not be "affected" by the persistent ones as long as you are using NPMSPEED(FAST) on your channels. They do not wait for a syncpoint. Paul Clarke could further discuss this with you I'm sure! Consider using channel pipelining as well. Q3. Related to Q1, yes you will eventually see I/O under certain circumstances... The certain circumstances have to do with arrival and processing rates of your non-persistent messages along with the setting of your DefaultQBufferSize. Each queue by default only has 64K of memory for non-persistent messages. So, if you're flooded with data or in the case of your 1GB of non-persistent data coming through, I'm sure you had some spill to the disk filesystem. You can augment the DefaultQBufferSize by setting it in the qm.ini in the TuningParameters stanza. HOWEVER, be sure to do this correctly!!! You need to set the parameter, restart the QM, define your queues that should have the extra large buffer, remove the stanza entry, and then restart the QM. You don't want to leave this set for all queues being created! The max size for the buffer is 1MB. Details about this parameter and usage can be found in the various performance report SupportPacs. Also, T.Rob was correct about in-memory transfer, but this only occurs between applications putting and getting from a queue simulataneously. The messages MUST NOT be IN_SYNCPOINT and they must be non-persistent. The applications will directly exchange the message in-memory and satisfy the MQ API calls on both ends (the PUT and the GET). Hope this helps! Gary ------------------------------ Gary J. Ward Senior Consulting Engineer Information Design, Inc. A Premier IBM Business Partner -------- Original Message -------- ==> From: "Wyatt, T. Rob" <[EMAIL PROTECTED]> ==> Date: Thu, 29 May 2003 11:28:59 -0500 Peter, I remember seeing a response from one of the IBM folks who said that some non-persistent message exchanges happen as an in-memory transfer. In general though, assume your non-persistent messages are written to disk but not logged. Persistent messages are written to disk AND to the log. -- T.Rob -----Original Message----- From: Potkay, Peter M (PLC, IT) [mailto:[EMAIL PROTECTED] Sent: Thursday, May 29, 2003 11:43 AM To: [EMAIL PROTECTED] Subject: How a MQSeries Hub does its thing with persistent / non-persist m essages Imagine if you will a Hub and Spoke architecture. QMHUB sits in the middle on a 2 CPU server. 1.3gig CPU, 2 gig RAM MQVersion 5.2.1 CSD05 QM1....QM20 are spokes. On QMHUB, there are no application queues. Only queue manager aliases and XMIT queues. There is a QMAlias for QM1 called QM1, and it directs messages to QM1.XMITQ, which is serviced by the SNDR channel off to QM1. This QMAlias / XMITQ / SNDR setup is present for every spoke. And there is also a RCVR channel as well from every spoke. On day 1 there is only non persistent message traffic for all the spokes. On day 2 a pair of the spokes starts exchanging persistent messages. Q1. On day 1, is there any data being written to disk by QMHUB as the messages fly thru? I assume no, since they are not persistent (but see Q3 below). Q2. On day 2, even though we have 2 CPUs, we still have only 1 QM, so I assume all the non persistent messages throughput must be affected by the persistent messages. My reasoning is, as the persistent messages go in and out of the QMAliases, and in and out of the XMIT queues, it has to "stop" and log, right? And if it has to stop and log, then it can't be handling the non persistent ones at the same time right? They have to wait? Q3. I then defined a local queue on QMHUB and used one of the spoke QMs to send non-persistent message to it. 1 GIG worth actually. Now these are not written to disk, cause they are not persistent, so where are they, in memory? I see the queue file grew by over a GIG, so doesn't that mean they are on disk, even though they are non persistent? More details.............. Batch Interval = 0 Batch Size = 50 Non Persistent Message Speed = Normal There is an MQCluster involved, which is why we have the QMAliases (can't cluster XMIT queues). I don't think this effects any of the answers to the above, but I can expound if need be. Peter Potkay MQSeries Specialist The Hartford Financial Services [EMAIL PROTECTED] x77906 IBM MQSeries Certified This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this communication and destroy all copies. Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive . Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive ********************************************************************** Click here to visit the Argos home page http://www.argos.co.uk The information contained in this message or any of its attachments may be privileged and confidential, and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you are not the intended addressee, any disclosure, reproduction, distribution, dissemination or use of this communication is not authorised. If you have received this message in error, please advise the sender by using your reply facility in your e-mail software. All messages sent and received by Argos Ltd are monitored for virus, high risk file extensions, and inappropriate content. As a result users should be aware that mail maybe accessed. ********************************************************************** Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
