The I/O discussion was held here a while ago and if memory serves me right. Persistent messages are written to the log from memory and then the application is told, GUD. If the messages are not taken off the queue quickly, and this might depend on both available space and time, they are then written to harden disk.
As for the single threading of messages through the queue manager. I will agree on Gary's premise that there are many QMGR related processes running. What now may come as interest is that the queues, and this is the way it was described to me (a couple of time by different people), are really a series of individual buckets split by priority and then by persisten and non-persistent. So it might be feasable that the non-persistent messages MAY be avaliable and processed seperatly where XMIT queues are involved. Who knows?? Only GOD and the Brits in Hursely!!
bobbee
From: Gary Ward <[EMAIL PROTECTED]> Reply-To: MQSeries List <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persistent messages Date: Thu, 29 May 2003 19:47:03 -0400
Peter,
I'll gladly defer to any IBM MQ Developers lurking on the list since this is getting really 'down and dirty', but in the meantime I'll give you another $.02...
Right about my answer to Q2... I guess I missed your point. Having read your follow-up, I would offer this theory. I think the queue manager MUST be able to do more than a single task at once. That's why it's made up of many individual processes which have dedicated tasks. I would think at any one moment there's a bunch of messages floating around in various states. If the messages are non-persistent, they're not logged and hence can be available immediately (if they're not IN SYNCPOINT with persistent messages). Non-persistent messages only get written to disk when there's not enough memory available to hold them on an individual queue. I'm not sure about persistent messages always going right to the queue file system, but my gut feeling is that they do.
If your non-persistent messages are 100K and you have your queues set at the 64KB default, I'm pretty sure they go right to disk. That's why you should tune that non-persistent message buffer. Hopefully something is already waiting for these non-persistent messages so they get handed off directly in the manner that T. Rob and I mentioned earlier in this thread. Then there's no I/O at all.
Hope this helps... any IBMer's want to comment????
-----Original Message----- From: MQSeries List [mailto:[EMAIL PROTECTED] Behalf Of Potkay, Peter M (PLC, IT) Sent: Thursday, May 29, 2003 6:11 PM To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi st m essages
Gary, I think you answer to Q2 pertains more to when a message is about to leave a server to go onto the next one. That setting tells the receiving side how soon it can have the Nonpersistent message in relation to the batch of messages coming across the channel.
I am curious about once the messages have already been accepted on the HUB. Whether they were persistent or not, regardless of the channel speed setting, at any given moment on our busy HUB, the QM finds itself with lots and lots of messages that it now has complete control over. As it routes them thru the QMAliases and the XMITQs, it has to "stop" and log the persistent messages. I feel that this activity must somehow also effect the non persistent ones as well, since the QM can only do one thing at a time, regardless how fast it does it. If it is busy logging a persistent message, it can't route a non persistent one at that exact moment, correct?
Regarding the queue buffer setting, if my messages are less than 64K, and because due to high activity all my channels in a particular SPOKE-HUB-SPOKE route are running, then a non persistent message would go in and out of each XMIT queue, in and out of each QM Alias queue and in and out of each application queue (assuming the app has an outstanding GET with wait) with no I/O to the disk?
What if the messages are 100K non persistent ones and the buffer setting is still at default? Are you saying that a non persistent message is still written to disk? If yes, to me that sounds like there is no reason to not use persistence always on any message larger than 64K. Surely that can't be the case! Or is it like T.Rob suggested: Nonpersistent gets written to disk, persistent gets written to disk AND log, for a double I/O???
-----Original Message----- From: Gary Ward [mailto:[EMAIL PROTECTED] Sent: Thursday, May 29, 2003 1:19 PM To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi st m essages
Let me throw in my $.02 - which is actually less in Euros lately ;)
Q1. There "could" be I/O to the queue filesystem... see more below
Q2. The non-persistent messages will not be "affected" by the persistent ones as long as you are using NPMSPEED(FAST) on your channels. They do not wait for a syncpoint. Paul Clarke could further discuss this with you I'm sure! Consider using channel pipelining as well.
Q3. Related to Q1, yes you will eventually see I/O under certain circumstances...
The certain circumstances have to do with arrival and processing rates of your non-persistent messages along with the setting of your DefaultQBufferSize. Each queue by default only has 64K of memory for non-persistent messages. So, if you're flooded with data or in the case of your 1GB of non-persistent data coming through, I'm sure you had some spill to the disk filesystem. You can augment the DefaultQBufferSize by setting it in the qm.ini in the TuningParameters stanza. HOWEVER, be sure to do this correctly!!! You need to set the parameter, restart the QM, define your queues that should have the extra large buffer, remove the stanza entry, and then restart the QM. You don't want to leave this set for all queues being created! The max size for the buffer is 1MB. Details about this parameter and usage can be found in the various performance report SupportPacs.
Also, T.Rob was correct about in-memory transfer, but this only occurs between applications putting and getting from a queue simulataneously. The messages MUST NOT be IN_SYNCPOINT and they must be non-persistent. The applications will directly exchange the message in-memory and satisfy the MQ API calls on both ends (the PUT and the GET).
Hope this helps! Gary
------------------------------ Gary J. Ward Senior Consulting Engineer Information Design, Inc. A Premier IBM Business Partner
-------- Original Message --------
==> From: "Wyatt, T. Rob" <[EMAIL PROTECTED]> ==> Date: Thu, 29 May 2003 11:28:59 -0500
Peter,
I remember seeing a response from one of the IBM folks who said that some non-persistent message exchanges happen as an in-memory transfer. In general though, assume your non-persistent messages are written to disk but not logged. Persistent messages are written to disk AND to the log.
-- T.Rob
-----Original Message----- From: Potkay, Peter M (PLC, IT) [mailto:[EMAIL PROTECTED] Sent: Thursday, May 29, 2003 11:43 AM To: [EMAIL PROTECTED] Subject: How a MQSeries Hub does its thing with persistent / non-persist m essages
Imagine if you will a Hub and Spoke architecture.
QMHUB sits in the middle on a 2 CPU server. 1.3gig CPU, 2 gig RAM MQVersion 5.2.1 CSD05 QM1....QM20 are spokes.
On QMHUB, there are no application queues. Only queue manager aliases and XMIT queues. There is a QMAlias for QM1 called QM1, and it directs messages to QM1.XMITQ, which is serviced by the SNDR channel off to QM1. This QMAlias / XMITQ / SNDR setup is present for every spoke. And there is also a RCVR channel as well from every spoke.
On day 1 there is only non persistent message traffic for all the spokes. On day 2 a pair of the spokes starts exchanging persistent messages.
Q1. On day 1, is there any data being written to disk by QMHUB as the messages fly thru? I assume no, since they are not persistent (but see Q3 below).
Q2. On day 2, even though we have 2 CPUs, we still have only 1 QM, so I assume all the non persistent messages throughput must be affected by the persistent messages. My reasoning is, as the persistent messages go in and out of the QMAliases, and in and out of the XMIT queues, it has to "stop" and log, right? And if it has to stop and log, then it can't be handling the non persistent ones at the same time right? They have to wait?
Q3. I then defined a local queue on QMHUB and used one of the spoke QMs to send non-persistent message to it. 1 GIG worth actually. Now these are not written to disk, cause they are not persistent, so where are they, in memory? I see the queue file grew by over a GIG, so doesn't that mean they are on disk, even though they are non persistent?
More details.............. Batch Interval = 0 Batch Size = 50 Non Persistent Message Speed = Normal There is an MQCluster involved, which is why we have the QMAliases (can't cluster XMIT queues). I don't think this effects any of the answers to the above, but I can expound if need be.
Peter Potkay MQSeries Specialist The Hartford Financial Services [EMAIL PROTECTED] x77906 IBM MQSeries Certified
This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this communication and destroy all copies.
Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive .
Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
_________________________________________________________________ The new MSN 8: smart spam protection and 2 months FREE* http://join.msn.com/?page=features/junkmail
Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
