Re: How a MQSeries Hub does its thing with persistent / non-persistent messages
> messages for this app so that changes to the SAN don't effect it. > > > > > > > > > > > > -Original Message- > > > From: John Scott [mailto:[EMAIL PROTECTED] > > > Sent: Friday, May 30, 2003 10:18 AM > > > To: [EMAIL PROTECTED] > > > Subject: Re: How a MQSeries Hub does its thing with persistent / > > > non-persi stent messages > > > > > > > > > This would be true for the smallest unit of execution (normally a > thread, > > > not a process). Thus if a thread of execution was committing data to > disk, > > > that thread would not continue until the commit was completed. However, > > > another thread within the same process would get CPU if it was able to > > > execute some code. > > > > > > It all depends on whether the MQ processes are multi-threaded or not. > > > > > > Regards > > > John. > > > > > > -Original Message- > > > From: Potkay, Peter M (PLC, IT) [mailto:[EMAIL PROTECTED]> > > > Sent: 30 May 2003 12:50 > > > To: [EMAIL PROTECTED] > > > Subject: Re: How a MQSeries Hub does its thing with persistent / > non-persi > > > stent messages > > > > > > > > > Gary, > > > About a queue manager, or any process that runs on any computer, > I > > > have always assumed, maybe incorrectly, that a CPU on a box can only do > > one > > > thing at a time. It may be incredibly fast, giving the illusion of many > > > things happening at once, but when you get right down to it, everything > > has > > > to wait its turn for the CPU to do its thing. > > > > > > And if the CPU is waiting to interact with the disk to write a > persistent > > > (or bigger than 64k non-persistent) message (I guess even if that > message > > is > > > going to blink in and out of a QM Alias or XMITQ???) then the CPU cant > go > > on > > > to handle non persistent messages. > > > > > > ??? > > > > > > > > > -Original Message- > > > From: Gary Ward [mailto:[EMAIL PROTECTED] > > > Sent: Thursday, May 29, 2003 7:47 PM > > > To: [EMAIL PROTECTED] > > > Subject: Re: How a MQSeries Hub does its thing with persistent / > > > non-persistent messages > > > > > > > > > Peter, > > > > > > I'll gladly defer to any IBM MQ Developers lurking on the list since > this > > is > > > getting really 'down and dirty', but in the meantime I'll give you > another > > > $.02... > > > > > > Right about my answer to Q2... I guess I missed your point. Having read > > > your follow-up, I would offer this theory. I think the queue manager > MUST > > > be able to do more than a single task at once. That's why it's made up > of > > > many individual processes which have dedicated tasks. I would think at > > any > > > one moment there's a bunch of messages floating around in various > states. > > If > > > the messages are non-persistent, they're not logged and hence can be > > > available immediately (if they're not IN SYNCPOINT with persistent > > > messages). Non-persistent messages only get written to disk when > there's> > > > not enough memory available to hold them on an individual queue. I'm > not > > > sure about persistent messages always going right to the queue file > > system, > > > but my gut feeling is that they do. > > > > > > If your non-persistent messages are 100K and you have your queues set at > > the > > > 64KB default, I'm pretty sure they go right to disk. That's why you > > should > > > tune that non-persistent message buffer. Hopefully something is already > > > waiting for these non-persistent messages so they get handed off > directly > > in > > > the manner that T. Rob and I mentioned earlier in this thread. Then > > there's > > > no I/O at all. > > > > > > Hope this helps... any IBMer's want to comment > > > > > > > > > -Original Message- > > > From: MQSeries List [mailto:[EMAIL PROTECTED] Behalf Of Potkay,> > > > Peter M (PLC, IT) > > > Sent: Thursday, May 29, 2003 6:11 PM > > > To: [EMAIL PROTECTED] > > > Subject: Re: How a MQSeries Hub does its thing with persistent / > non-persi > > > st m essages &g
Re: How a MQSeries Hub does its thing with persistent / non-persistent messages
erse affect with respect to it. Point #2. If the disk is temporarily unavailable, then persistent messages are effected, and due to point #1, non persistent ones are effected as well. I agree (in a contrary way): If tasks handling non-persistent messages are dependent on disk, then they may be adversely affected by the unavailable disk. If tasks handling non-persistent messages are not dependent on disk, then they may be favorably affected by the unavailable disk! This occurs because other tasks waiting on disk are not competing for CPU. > -Original Message- > From: Potkay, Peter M (PLC, IT) [SMTP:[EMAIL PROTECTED] > Sent: Friday, May 30, 2003 1:01 PM > To: [EMAIL PROTECTED] > Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi stent messages > > Hi Dennis. > > I agree that non-persistent and persistent messages in a batch will be > available at the same time. But my batches are of 1 or 2 messages. Since my > BATCHINT is zero, and the channel can keep the XMIT queue at zero, I doubt I > am incurring any performance hit. 1 or 2 messages are sent across, mybe > message #1 is a nonpersistent that is held back until the batch completes, > but the batch completes almost immediately, since no more message have yet > arrived and the BATCHINT expires. At this point I am still not convinced, in > my scenario, that this setting has an effect for the above reason.> > > It has also been mentioned more than once to make a separate channel for > persistent vs. non-persistent messages. I don't see how that makes a > difference. So I make 2 separate channels from SPOKEQM1 to HUBQM. As far as > the hub and its receiving MCAs are concerned, big deal. The HUB already had > 50 other receiving MCAs and channels for all the other spokes. If the HUB QM > is slowing down because its SAN is temporarily unavailable, does it really > matter if you have 1,2 or 20 channels between a particular spoke and the > HUB? The HUB still has to deal with the other 50 spokes, and if they are > sending persistent messages, then the HUBQM has to service them as well, and > it makes no diff that you bothered to separate the channels from any one > particular spoke. All these Receiving MCAs have to compete for 1 or 2 CPUs, > and if any of those CPUs are busy waiting because the SAN is N/A, then a > whole bunch of MCAs are effected. > > > I wonder what everyone thinks of the following: > > Point #1. If a server is dealing with persistent and non persistent > messages, the persistent ones have to negatively effect the non persistent > ones, at a hardware level (disk and CPU). It does not matter whether you > have separate QMs on that server split between persistent and non > persistent. Both QMs share the CPU and disk. It does not matter whether you > have separate channels or not. All MCAs share the same CPU and disk. > > Point #2. If the disk is temporarily unavailable, then persistent messages > are effected, and due to point #1, non persistent ones are effected as well. > > > > > > > > -Original Message- > From: Miller, Dennis [mailto:[EMAIL PROTECTED] > Sent: Friday, May 30, 2003 1:55 PM > To: [EMAIL PROTECTED] > Subject: Re: How a MQSeries Hub does its thing with persistent / > non-persistent messages > > > Peter, > If you use NPMSPEED=normal, non-persistent and persistent messages in the > same batch all become available at the same time. You can infer from that, > that performance of non-persistent messages is dependent on I/O for > persistent messages, though I believe it is more likely to be the > (synchronous) log I/O rather than (asynchronous) queue I/O. > > Segregating persistent and non-persistent messages on different channels > relieves the above dependency. If one channel is waiting for the log, > another should be able to continue transmitting messages until such time as > it also needs the log or until the qmgr decides a checkpoint is in order. > > > Bottom line: > If your logs are on the SAN, there is a distinct probability that your spoke > channels will take a break when that "device" is not available. > > my .02 sense (pun intentional). > > > > -Original Message- > > From: Potkay, Peter M (PLC, IT) [SMTP:[EMAIL PROTECTED] > > Sent: Friday, May 30, 2003 7:21 AM > > To: [EMAIL PROTECTED] > > Subject: Re: How a MQSeries Hub does its thing with persistent / > non-persi stent messages > > > > So here is my real question, which is what makes me wonder exactly how a > QM > > handles messages. > > > > Our HUB server is using Veritas. The disk that is being written to > (whenever > > that may be) is a
Re: How a MQSeries Hub does its thing with persistent / non-persistent messages
Peter, If you use NPMSPEED=normal, non-persistent and persistent messages in the same batch all become available at the same time. You can infer from that, that performance of non-persistent messages is dependent on I/O for persistent messages, though I believe it is more likely to be the (synchronous) log I/O rather than (asynchronous) queue I/O. Segregating persistent and non-persistent messages on different channels relieves the above dependency. If one channel is waiting for the log, another should be able to continue transmitting messages until such time as it also needs the log or until the qmgr decides a checkpoint is in order. Bottom line: If your logs are on the SAN, there is a distinct probability that your spoke channels will take a break when that "device" is not available. my .02 sense (pun intentional). > -Original Message- > From: Potkay, Peter M (PLC, IT) [SMTP:[EMAIL PROTECTED] > Sent: Friday, May 30, 2003 7:21 AM > To: [EMAIL PROTECTED] > Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi > stent messages > > So here is my real question, which is what makes me wonder exactly how a QM > handles messages. > > Our HUB server is using Veritas. The disk that is being written to (whenever > that may be) is actually on the Storage Area Network (SAN). > > The HUB is also clustered with 2 queue managers dedicated to MQSI. The HUB > acts simply as a gateway queue manager for this MQSI cluster. THE MQSI boxes > are in 2 separate locations, with Veritas, and both again also write to the > SAN. > > > Whenever we make bin changes to the SAN, that change ripples across the > fabric, making the SAN unavailable for a tiny bit of time. > > Now, we have an app that is counting milliseconds in its roundtrip of the > message. This message starts on one of the spokes, comes to the HUB, is > round robined to one of the MQSI boxes, the processed message comes back to > the HUB, which then sends it down to the receiving spoke. It processes the > message, sends it back to the hub, round robined into MQSI for processing > the reply, the processed reply goes back to the hub, which then sends the > reply back to the originating spoke. For 99.99% of the time, this roundtrip > takes under 500 milliseconds. The app waits up to 2000 milliseconds for the > reply. The messages are non persistent and about 25K in size. > > Whenever the bin changes to the SAN take place, we start getting messages > that take longer than 2000 milliseconds, and now we have orphaned replies. > These are non persistent messages that are under 64K, so why does a change > that makes the disk unavailable cause these messages to slow down? My guess > is that the persistent messages the HUB is processing at the same time (or > the >64K Nonpersistent ones) must somehow be effecting the performance of > the non persistent ones. And I also assume that channel speed has nothing to > do with this. > > So the angle I am after here is how can I increase the performance of my > messages for this app so that changes to the SAN don't effect it. > > > > -Original Message- > From: John Scott [mailto:[EMAIL PROTECTED] > Sent: Friday, May 30, 2003 10:18 AM > To: [EMAIL PROTECTED] > Subject: Re: How a MQSeries Hub does its thing with persistent / > non-persi stent messages > > > This would be true for the smallest unit of execution (normally a thread, > not a process). Thus if a thread of execution was committing data to disk, > that thread would not continue until the commit was completed. However, > another thread within the same process would get CPU if it was able to > execute some code. > > It all depends on whether the MQ processes are multi-threaded or not. > > Regards > John. > > -Original Message- > From: Potkay, Peter M (PLC, IT) [mailto:[EMAIL PROTECTED]> > Sent: 30 May 2003 12:50 > To: [EMAIL PROTECTED] > Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi > stent messages > > > Gary, > About a queue manager, or any process that runs on any computer, I > have always assumed, maybe incorrectly, that a CPU on a box can only do one > thing at a time. It may be incredibly fast, giving the illusion of many > things happening at once, but when you get right down to it, everything has > to wait its turn for the CPU to do its thing. > > And if the CPU is waiting to interact with the disk to write a persistent > (or bigger than 64k non-persistent) message (I guess even if that message is > going to blink in and out of a QM Alias or XMITQ???) then the CPU cant go on > to handle non persistent messages. > > ??? > > > -Original Me
Re: How a MQSeries Hub does its thing with persistent / non-persistent messages
One point to back up here. The I/O discussion was held here a while ago and if memory serves me right. Persistent messages are written to the log from memory and then the application is told, GUD. If the messages are not taken off the queue quickly, and this might depend on both available space and time, they are then written to harden disk. As for the single threading of messages through the queue manager. I will agree on Gary's premise that there are many QMGR related processes running. What now may come as interest is that the queues, and this is the way it was described to me (a couple of time by different people), are really a series of individual buckets split by priority and then by persisten and non-persistent. So it might be feasable that the non-persistent messages MAY be avaliable and processed seperatly where XMIT queues are involved. Who knows?? Only GOD and the Brits in Hursely!! bobbee From: Gary Ward <[EMAIL PROTECTED]> Reply-To: MQSeries List <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persistent messages Date: Thu, 29 May 2003 19:47:03 -0400 Peter, I'll gladly defer to any IBM MQ Developers lurking on the list since this is getting really 'down and dirty', but in the meantime I'll give you another $.02... Right about my answer to Q2... I guess I missed your point. Having read your follow-up, I would offer this theory. I think the queue manager MUST be able to do more than a single task at once. That's why it's made up of many individual processes which have dedicated tasks. I would think at any one moment there's a bunch of messages floating around in various states. If the messages are non-persistent, they're not logged and hence can be available immediately (if they're not IN SYNCPOINT with persistent messages). Non-persistent messages only get written to disk when there's not enough memory available to hold them on an individual queue. I'm not sure about persistent messages always going right to the queue file system, but my gut feeling is that they do. If your non-persistent messages are 100K and you have your queues set at the 64KB default, I'm pretty sure they go right to disk. That's why you should tune that non-persistent message buffer. Hopefully something is already waiting for these non-persistent messages so they get handed off directly in the manner that T. Rob and I mentioned earlier in this thread. Then there's no I/O at all. Hope this helps... any IBMer's want to comment -Original Message- From: MQSeries List [mailto:[EMAIL PROTECTED] Behalf Of Potkay, Peter M (PLC, IT) Sent: Thursday, May 29, 2003 6:11 PM To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi st m essages Gary, I think you answer to Q2 pertains more to when a message is about to leave a server to go onto the next one. That setting tells the receiving side how soon it can have the Nonpersistent message in relation to the batch of messages coming across the channel. I am curious about once the messages have already been accepted on the HUB. Whether they were persistent or not, regardless of the channel speed setting, at any given moment on our busy HUB, the QM finds itself with lots and lots of messages that it now has complete control over. As it routes them thru the QMAliases and the XMITQs, it has to "stop" and log the persistent messages. I feel that this activity must somehow also effect the non persistent ones as well, since the QM can only do one thing at a time, regardless how fast it does it. If it is busy logging a persistent message, it can't route a non persistent one at that exact moment, correct? Regarding the queue buffer setting, if my messages are less than 64K, and because due to high activity all my channels in a particular SPOKE-HUB-SPOKE route are running, then a non persistent message would go in and out of each XMIT queue, in and out of each QM Alias queue and in and out of each application queue (assuming the app has an outstanding GET with wait) with no I/O to the disk? What if the messages are 100K non persistent ones and the buffer setting is still at default? Are you saying that a non persistent message is still written to disk? If yes, to me that sounds like there is no reason to not use persistence always on any message larger than 64K. Surely that can't be the case! Or is it like T.Rob suggested: Nonpersistent gets written to disk, persistent gets written to disk AND log, for a double I/O??? -Original Message- From: Gary Ward [mailto:[EMAIL PROTECTED] Sent: Thursday, May 29, 2003 1:19 PM To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi st m essages Let me throw in my $.02 - which is actually less in Euros lately ;) Q1. There "could" be I/O to the que
Re: How a MQSeries Hub does its thing with persistent / non-persistent messages
Hi... not an IBMer, but want to comment. The following are my thoughts
based on behaviour I have seen on my queue managers, not based on the code.
Q1. You will see log I/O in your environment, because you are running
NPMSpeed normal. This causes the channel sequence numbers to be hardened,
which causes queue file (on the SYSTEM.CHANNEL.SYNCQ) and log I/O. It also
causes non-persistent messages to be written inside syncpoint control by
the channel agents, so you can't get the performance advantages of direct
message copying between the agents. I have recently come to understand that
MQ also uses scratchpad files to track channel info, and there is I/O to
these files as well.
If you were to change to NPMSPEED(fast), then you should reduce both log
and queue file I/O in case 1.
However, when you have persistent and non-persistent messages sharing an
NPMSpeed(Fast) channel, each persistent message will start a batch, and
non-persistent messages can get sent in the batch, and so the
non-persistent messages get caught up in the synpoints (but their data is
not logged). This delays their availability on the target queue.
Queue file I/O should generally asynchronous (look on it like paging to
virtual memory) whereas logging writes are (at least sometimes)
synchronous. (Data Logging is probably async, but syncpoint logging has to
be synchronous I/O). Think of it just like a database logger. This means
that not all I/O is equally bad.
I would suggest that to get maximum performance, you need to separate your
persistent and non-persistent traffic, and run NPMSpeed(Fast).
This is difficult, as you need to double your infrastructure, and you won't
be able to use default transmission queues (I have always thought they were
a bad idea, but that is just me). You define 2 channels from each leaf to
the hub, and from the hub to each leaf, one for persistent and one for
non-persistent. You need to have QMgrAliases everywhere, with different
names for accesses to a queue manager with persistent vs non-persistent
messages. You need to have additional ReplyToQ remote queue definitions so
that replies come back via the correct aliases, as otherwise all of the
replies will be routed via the same channel. You also need to define
QRemotes for all of the queues at every queue manager, rather than just on
the hub, as default transmission queues won't separate the persistent and
non-persistent traffic. If persistent and non-persistent traffic hits the
same queues, you will need to define 2 QRemotes, to direct the traffic to
the correct transmistion queues.
However, I don't think the situation in Q2 is as bad as you assume. Lots of
the I/O which a queue manager does is asynchronous, and there are separate
processes (and threads) handling different bits, so not everything waits
just because 1 thread is blocked on a synchronous log I/O.
On Q3, the data will be written to disk, but the writes can be
asynchronous. The queue file will grow to contain all the messages unless
you have a process removing them as they are added.
Regards... Neil Casey
|-+>
| | Gary Ward|
| | <[EMAIL PROTECTED]|
| | A.NET> |
| | Sent by: MQSeries|
| | List |
| | <[EMAIL PROTECTED]|
| | n.AC.AT> |
| ||
| ||
| | 30/05/2003 09:47 |
| | Please respond to|
| | MQSeries List|
| ||
|-+>
>--|
|
|
| To: [EMAIL PROTECTED]
|
| cc:
|
| Subject: Re: How a MQSeries Hub does its thing with persistent /
non-persistent messages |
>--|
Peter,
I'll gladly defer to any IBM MQ Developers lurking on the list since this
is
getting really 'down and dirty', but in the meantime I'll give you another
$.02...
Right about my answer to Q2... I guess I missed your point. Having read
your follow-up, I would offer this theory. I think the queue manager MUST
be able to do more than a single task at once. That's why it's made up of
many individual processes which have dedicated tasks. I would think at any
one moment there's a bunch of messages floating aroun
Re: How a MQSeries Hub does its thing with persistent / non-persistent messages
Peter, I'll gladly defer to any IBM MQ Developers lurking on the list since this is getting really 'down and dirty', but in the meantime I'll give you another $.02... Right about my answer to Q2... I guess I missed your point. Having read your follow-up, I would offer this theory. I think the queue manager MUST be able to do more than a single task at once. That's why it's made up of many individual processes which have dedicated tasks. I would think at any one moment there's a bunch of messages floating around in various states. If the messages are non-persistent, they're not logged and hence can be available immediately (if they're not IN SYNCPOINT with persistent messages). Non-persistent messages only get written to disk when there's not enough memory available to hold them on an individual queue. I'm not sure about persistent messages always going right to the queue file system, but my gut feeling is that they do. If your non-persistent messages are 100K and you have your queues set at the 64KB default, I'm pretty sure they go right to disk. That's why you should tune that non-persistent message buffer. Hopefully something is already waiting for these non-persistent messages so they get handed off directly in the manner that T. Rob and I mentioned earlier in this thread. Then there's no I/O at all. Hope this helps... any IBMer's want to comment -Original Message- From: MQSeries List [mailto:[EMAIL PROTECTED] Behalf Of Potkay, Peter M (PLC, IT) Sent: Thursday, May 29, 2003 6:11 PM To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi st m essages Gary, I think you answer to Q2 pertains more to when a message is about to leave a server to go onto the next one. That setting tells the receiving side how soon it can have the Nonpersistent message in relation to the batch of messages coming across the channel. I am curious about once the messages have already been accepted on the HUB. Whether they were persistent or not, regardless of the channel speed setting, at any given moment on our busy HUB, the QM finds itself with lots and lots of messages that it now has complete control over. As it routes them thru the QMAliases and the XMITQs, it has to "stop" and log the persistent messages. I feel that this activity must somehow also effect the non persistent ones as well, since the QM can only do one thing at a time, regardless how fast it does it. If it is busy logging a persistent message, it can't route a non persistent one at that exact moment, correct? Regarding the queue buffer setting, if my messages are less than 64K, and because due to high activity all my channels in a particular SPOKE-HUB-SPOKE route are running, then a non persistent message would go in and out of each XMIT queue, in and out of each QM Alias queue and in and out of each application queue (assuming the app has an outstanding GET with wait) with no I/O to the disk? What if the messages are 100K non persistent ones and the buffer setting is still at default? Are you saying that a non persistent message is still written to disk? If yes, to me that sounds like there is no reason to not use persistence always on any message larger than 64K. Surely that can't be the case! Or is it like T.Rob suggested: Nonpersistent gets written to disk, persistent gets written to disk AND log, for a double I/O??? -Original Message- From: Gary Ward [mailto:[EMAIL PROTECTED] Sent: Thursday, May 29, 2003 1:19 PM To: [EMAIL PROTECTED] Subject: Re: How a MQSeries Hub does its thing with persistent / non-persi st m essages Let me throw in my $.02 - which is actually less in Euros lately ;) Q1. There "could" be I/O to the queue filesystem... see more below Q2. The non-persistent messages will not be "affected" by the persistent ones as long as you are using NPMSPEED(FAST) on your channels. They do not wait for a syncpoint. Paul Clarke could further discuss this with you I'm sure! Consider using channel pipelining as well. Q3. Related to Q1, yes you will eventually see I/O under certain circumstances... The certain circumstances have to do with arrival and processing rates of your non-persistent messages along with the setting of your DefaultQBufferSize. Each queue by default only has 64K of memory for non-persistent messages. So, if you're flooded with data or in the case of your 1GB of non-persistent data coming through, I'm sure you had some spill to the disk filesystem. You can augment the DefaultQBufferSize by setting it in the qm.ini in the TuningParameters stanza. HOWEVER, be sure to do this correctly!!! You need to set the parameter, restart the QM, define your queues that should have the extra large buffer, remove the stanza entry, and then restart the QM. You don't want to leave this set for all queues being created! The max size for the buffer is 1MB. Details about this parameter and usage can be found in the various performance report S
