Peter, If you use NPMSPEED=normal, non-persistent and persistent messages in the same batch all become available at the same time. You can infer from that, that performance of non-persistent messages is dependent on I/O for persistent messages, though I believe it is more likely to be the (synchronous) log I/O rather than (asynchronous) queue I/O.
Segregating persistent and non-persistent messages on different channels relieves the above dependency. If one channel is waiting for the log, another should be able to continue transmitting messages until such time as it also needs the log or until the qmgr decides a checkpoint is in order. Bottom line: If your logs are on the SAN, there is a distinct probability that your spoke channels will take a break when that "device" is not available. my .02 sense (pun intentional). > -----Original Message----- > From: Potkay, Peter M (PLC, IT) [SMTP:[EMAIL PROTECTED] > Sent: Friday, May 30, 2003 7:21 AM > To: [EMAIL PROTECTED] > Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi > stent messages > > So here is my real question, which is what makes me wonder exactly how a QM > handles messages. > > Our HUB server is using Veritas. The disk that is being written to (whenever > that may be) is actually on the Storage Area Network (SAN). > > The HUB is also clustered with 2 queue managers dedicated to MQSI. The HUB > acts simply as a gateway queue manager for this MQSI cluster. THE MQSI boxes > are in 2 separate locations, with Veritas, and both again also write to the > SAN. > > > Whenever we make bin changes to the SAN, that change ripples across the > fabric, making the SAN unavailable for a tiny bit of time. > > Now, we have an app that is counting milliseconds in its roundtrip of the > message. This message starts on one of the spokes, comes to the HUB, is > round robined to one of the MQSI boxes, the processed message comes back to > the HUB, which then sends it down to the receiving spoke. It processes the > message, sends it back to the hub, round robined into MQSI for processing > the reply, the processed reply goes back to the hub, which then sends the > reply back to the originating spoke. For 99.99% of the time, this roundtrip > takes under 500 milliseconds. The app waits up to 2000 milliseconds for the > reply. The messages are non persistent and about 25K in size. > > Whenever the bin changes to the SAN take place, we start getting messages > that take longer than 2000 milliseconds, and now we have orphaned replies. > These are non persistent messages that are under 64K, so why does a change > that makes the disk unavailable cause these messages to slow down? My guess > is that the persistent messages the HUB is processing at the same time (or > the >64K Nonpersistent ones) must somehow be effecting the performance of > the non persistent ones. And I also assume that channel speed has nothing to > do with this. > > So the angle I am after here is how can I increase the performance of my > messages for this app so that changes to the SAN don't effect it. > > > > -----Original Message----- > From: John Scott [mailto:[EMAIL PROTECTED] > Sent: Friday, May 30, 2003 10:18 AM > To: [EMAIL PROTECTED] > Subject: Re: How a MQSeries Hub does its thing with persistent / > non-persi stent messages > > > This would be true for the smallest unit of execution (normally a thread, > not a process). Thus if a thread of execution was committing data to disk, > that thread would not continue until the commit was completed. However, > another thread within the same process would get CPU if it was able to > execute some code. > > It all depends on whether the MQ processes are multi-threaded or not. > > Regards > John. > > -----Original Message----- > From: Potkay, Peter M (PLC, IT) [mailto:[EMAIL PROTECTED]> > Sent: 30 May 2003 12:50 > To: [EMAIL PROTECTED] > Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi > stent messages > > > Gary, > About a queue manager, or any process that runs on any computer, I > have always assumed, maybe incorrectly, that a CPU on a box can only do one > thing at a time. It may be incredibly fast, giving the illusion of many > things happening at once, but when you get right down to it, everything has > to wait its turn for the CPU to do its thing. > > And if the CPU is waiting to interact with the disk to write a persistent > (or bigger than 64k non-persistent) message (I guess even if that message is > going to blink in and out of a QM Alias or XMITQ???) then the CPU cant go on > to handle non persistent messages. > > ??? > > > -----Original Message----- > From: Gary Ward [mailto:[EMAIL PROTECTED] > Sent: Thursday, May 29, 2003 7:47 PM > To: [EMAIL PROTECTED] > Subject: Re: How a MQSeries Hub does its thing with persistent / > non-persistent messages > > > Peter, > > I'll gladly defer to any IBM MQ Developers lurking on the list since this is > getting really 'down and dirty', but in the meantime I'll give you another > $.02... > > Right about my answer to Q2... I guess I missed your point. Having read > your follow-up, I would offer this theory. I think the queue manager MUST > be able to do more than a single task at once. That's why it's made up of > many individual processes which have dedicated tasks. I would think at any > one moment there's a bunch of messages floating around in various states. If > the messages are non-persistent, they're not logged and hence can be > available immediately (if they're not IN SYNCPOINT with persistent > messages). Non-persistent messages only get written to disk when there's > not enough memory available to hold them on an individual queue. I'm not > sure about persistent messages always going right to the queue file system, > but my gut feeling is that they do. > > If your non-persistent messages are 100K and you have your queues set at the > 64KB default, I'm pretty sure they go right to disk. That's why you should > tune that non-persistent message buffer. Hopefully something is already > waiting for these non-persistent messages so they get handed off directly in > the manner that T. Rob and I mentioned earlier in this thread. Then there's > no I/O at all. > > Hope this helps... any IBMer's want to comment???? > > > -----Original Message----- > From: MQSeries List [mailto:[EMAIL PROTECTED] Behalf Of Potkay, > Peter M (PLC, IT) > Sent: Thursday, May 29, 2003 6:11 PM > To: [EMAIL PROTECTED] > Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi > st m essages > > > Gary, > I think you answer to Q2 pertains more to when a message is about to > leave a server to go onto the next one. That setting tells the receiving > side how soon it can have the Nonpersistent message in relation to the batch > of messages coming across the channel. > > I am curious about once the messages have already been accepted on the HUB. > Whether they were persistent or not, regardless of the channel speed > setting, at any given moment on our busy HUB, the QM finds itself with lots > and lots of messages that it now has complete control over. As it routes > them thru the QMAliases and the XMITQs, it has to "stop" and log the > persistent messages. I feel that this activity must somehow also effect the > non persistent ones as well, since the QM can only do one thing at a time, > regardless how fast it does it. If it is busy logging a persistent message, > it can't route a non persistent one at that exact moment, correct? > > > Regarding the queue buffer setting, if my messages are less than 64K, and > because due to high activity all my channels in a particular SPOKE-HUB-SPOKE > route are running, then a non persistent message would go in and out of each> > XMIT queue, in and out of each QM Alias queue and in and out of each > application queue (assuming the app has an outstanding GET with wait) with > no I/O to the disk? > > What if the messages are 100K non persistent ones and the buffer setting is > still at default? Are you saying that a non persistent message is still > written to disk? If yes, to me that sounds like there is no reason to not > use persistence always on any message larger than 64K. Surely that can't be > the case! Or is it like T.Rob suggested: Nonpersistent gets written to > disk, persistent gets written to disk AND log, for a double I/O??? > > > > -----Original Message----- > From: Gary Ward [mailto:[EMAIL PROTECTED] > Sent: Thursday, May 29, 2003 1:19 PM > To: [EMAIL PROTECTED] > Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi > st m essages > > > Let me throw in my $.02 - which is actually less in Euros lately ;) > > Q1. There "could" be I/O to the queue filesystem... see more below > > Q2. The non-persistent messages will not be "affected" by the persistent > ones as long as you are using NPMSPEED(FAST) on your channels. They do not > wait for a syncpoint. Paul Clarke could further discuss this with you I'm > sure! Consider using channel pipelining as well. > > Q3. Related to Q1, yes you will eventually see I/O under certain > circumstances... > > The certain circumstances have to do with arrival and processing rates of > your non-persistent messages along with the setting of your > DefaultQBufferSize. Each queue by default only has 64K of memory for > non-persistent messages. So, if you're flooded with data or in the case of > your 1GB of non-persistent data coming through, I'm sure you had some spill > to the disk filesystem. You can augment the DefaultQBufferSize by setting > it in the qm.ini in the TuningParameters stanza. HOWEVER, be sure to do > this correctly!!! You need to set the parameter, restart the QM, define your > queues that should have the extra large buffer, remove the stanza entry, and > then restart the QM. You don't want to leave this set for all queues being > created! The max size for the buffer is 1MB. Details about this parameter > and usage can be found in the various performance report SupportPacs. > > Also, T.Rob was correct about in-memory transfer, but this only occurs > between applications putting and getting from a queue simulataneously. The > messages MUST NOT be IN_SYNCPOINT and they must be non-persistent. The > applications will directly exchange the message in-memory and satisfy the MQ > API calls on both ends (the PUT and the GET). > > Hope this helps! > Gary > > ------------------------------ > Gary J. Ward > Senior Consulting Engineer > Information Design, Inc. > A Premier IBM Business Partner > > > -------- Original Message -------- > > ==> From: "Wyatt, T. Rob" <[EMAIL PROTECTED]> > ==> Date: Thu, 29 May 2003 11:28:59 -0500 > > Peter, > > I remember seeing a response from one of the IBM folks who said that some > non-persistent message exchanges happen as an in-memory transfer. In > general though, assume your non-persistent messages are written to disk but > not logged. Persistent messages are written to disk AND to the log. > > -- T.Rob > > -----Original Message----- From: Potkay, Peter M (PLC, IT) > [mailto:[EMAIL PROTECTED] Sent: Thursday, May 29, 2003 11:43 AM > To: [EMAIL PROTECTED] Subject: How a MQSeries Hub does its thing with > persistent / non-persist m essages > > Imagine if you will a Hub and Spoke architecture. > > QMHUB sits in the middle on a 2 CPU server. 1.3gig CPU, 2 gig RAM MQVersion > 5.2.1 CSD05 QM1....QM20 are spokes. > > On QMHUB, there are no application queues. Only queue manager aliases and > XMIT queues. There is a QMAlias for QM1 called QM1, and it directs messages > to QM1.XMITQ, which is serviced by the SNDR channel off to QM1. This QMAlias> > / XMITQ / SNDR setup is present for every spoke. And there is also a RCVR > channel as well from every spoke. > > On day 1 there is only non persistent message traffic for all the spokes. > On day 2 a pair of the spokes starts exchanging persistent messages. > > Q1. On day 1, is there any data being written to disk by QMHUB as the > messages fly thru? I assume no, since they are not persistent (but see Q3 > below). > > Q2. On day 2, even though we have 2 CPUs, we still have only 1 QM, so I > assume all the non persistent messages throughput must be affected by the > persistent messages. My reasoning is, as the persistent messages go in and > out of the QMAliases, and in and out of the XMIT queues, it has to "stop" > and log, right? And if it has to stop and log, then it can't be handling the > non persistent ones at the same time right? They have to wait? > > Q3. I then defined a local queue on QMHUB and used one of the spoke QMs to > send non-persistent message to it. 1 GIG worth actually. Now these are not > written to disk, cause they are not persistent, so where are they, in > memory? I see the queue file grew by over a GIG, so doesn't that mean they > are on disk, even though they are non persistent? > > > > More details.............. Batch Interval = 0 Batch Size = 50 Non > Persistent Message Speed = Normal There is an MQCluster involved, which is > why we have the QMAliases (can't cluster XMIT queues). I don't think this > effects any of the answers to the above, but I can expound if need be. > > > > Peter Potkay MQSeries Specialist The Hartford Financial Services > [EMAIL PROTECTED] x77906 IBM MQSeries Certified > > > > This communication, including attachments, is for the exclusive use of > addressee and may contain proprietary, confidential or privileged > information. If you are not the intended recipient, any use, copying, > disclosure, dissemination or distribution is strictly prohibited. If you are > not the intended recipient, please notify the sender immediately by return > email and delete this communication and destroy all copies. > > Instructions for managing your mailing list subscription are provided in the > Listserv General Users Guide available at http://www.lsoft.com > Archive: http://vm.akh-wien.ac.at/MQSeries.archive > > Instructions for managing your mailing list subscription are provided in the > Listserv General Users Guide available at http://www.lsoft.com > Archive: http://vm.akh-wien.ac.at/MQSeries.archive . > > Instructions for managing your mailing list subscription are provided in the > Listserv General Users Guide available at http://www.lsoft.com > Archive: http://vm.akh-wien.ac.at/MQSeries.archive > > Instructions for managing your mailing list subscription are provided in the > Listserv General Users Guide available at http://www.lsoft.com > Archive: http://vm.akh-wien.ac.at/MQSeries.archive > > Instructions for managing your mailing list subscription are provided in the > Listserv General Users Guide available at http://www.lsoft.com > Archive: http://vm.akh-wien.ac.at/MQSeries.archive > > Instructions for managing your mailing list subscription are provided in the > Listserv General Users Guide available at http://www.lsoft.com > Archive: http://vm.akh-wien.ac.at/MQSeries.archive > > > ********************************************************************** > > Click here to visit the Argos home page http://www.argos.co.uk > > The information contained in this message or any of its attachments may be > privileged and confidential, and is intended exclusively for the addressee. > The views expressed may not be official policy, but the personal views of > the originator. > If you are not the intended addressee, any disclosure, reproduction, > distribution, dissemination or use of this communication is not authorised. > If you have received this message in error, please advise the sender by> > using your reply facility in your e-mail software. > All messages sent and received by Argos Ltd are monitored for virus, high > risk file extensions, and inappropriate content. As a result users should be > aware that mail maybe accessed. > > ********************************************************************** > > Instructions for managing your mailing list subscription are provided in > the Listserv General Users Guide available at http://www.lsoft.com > Archive: http://vm.akh-wien.ac.at/MQSeries.archive > > Instructions for managing your mailing list subscription are provided in > the Listserv General Users Guide available at http://www.lsoft.com > Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
