Re: Message Sequence Problem
Try very very hard to get rid of message affinity wherever possible. Makes scaling up much easier. If you can't then Dennis's method is as good as any. If using clustering your putting app will need to use MQOO_BIND_ON_OPEN to avoid messages in a given batch going to different queue managers. Also beware of Priority and MQGET's when using browsing. Regards Tim A -Original Message- From: Miller, Dennis [mailto:[EMAIL PROTECTED] Sent: Thursday, 11 November 2004 10:40 AM To: [EMAIL PROTECTED] Subject: Re: Message Sequence Problem Message order can be preserved by following certain conditions, most which are detailed in the various programming guides. Any designs depending on those conditions, however, have potential to break if those conditions change, either within your enterprise or in a future product release. Down the road, who will remember that some application developed 5 years ago may go south if another channel is added between two qmgrs or workload increases and two programs are needed to send messages. On the otherhand, it's trivial to use logical message groups or a msgid/correlid mechanism to unconditionally preserve message order. Take the time now and save problems later. A simple and elegant method is: On put, let MQ generate unique msgid's On first message of sequence, put a special value in correlid On each message after the first, put msgid of the previous message in the correlid Send trailer message to indicate no more messages in the sequence. On first get, retrieve first message with special value in correlid On subsequent gets, use msgid from previous msg to selectively retrieve next message Continue until trailer message is encountered Regards, Dennis -Original Message- From: Wesley Shaw [mailto:[EMAIL PROTECTED] Sent: Wednesday, November 10, 2004 12:38 PM To: [EMAIL PROTECTED] Subject: Message Sequence Problem The messages being processed are not in order when they are processed from the MVS local queue, but processing the same request through a local queue on the Unix box, the messages are processed in order. What are the rules for processing of message in order? Seems like all the same Priority would process FIFO. Can we guarentee the order of the message to be got same as they were put ? MVS view of 416345 Local Queue All 153 bytes except the * record which is 9 bytes, all messages Priority 1 with persistence Transmit queue MSGDLVSQ(PRIORITY), Local Unix queue MSGDLVSQ(PRIORITY) MVS Local queue Message delivery sequence FIFO The first list is a local queue on MVS which is remote to the putting application. 00416345RECC,CMPL03CMPL 20041108115902200411081225512004 00416345RECC 03CMPL 20041108115902200411081225512004 00416345RECC 02ARRV 20041108115902200411081225512004 00416345RECC 02ENRT 20041108115902200411081225512004 00416345RECC 02DISP 2004110811590220041108122551 00416345RECC 02ASSN20041108115902 00416345* 00416345RECC 01UNAS This second view is a local queue on Unix of the same put process as above. Unix view of 416345 9 00416345RECC CMPL03CMPL ' 10 00416345RECC 03CMPL 11 00416345RECC 02ARRV 12 00416345RECC 02ENRT 13 00416345RECC 02DISP 14 00416345RECC 02ASSN 15 00416345RECC 01UNAS 16 00416345* Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive This email and any attachments may contain privileged and confidential information and are intended for the named addressee only. If you have received this e-mail in error, please notify the sender and delete this e-mail immediately. Any confidentiality, privilege or copyright is not waived or lost because this e-mail has been sent to you in error. It is your responsibility to check this e-mail and any attachments for viruses. No warranty is made that this material is free from computer virus or any other defect or error. Any loss/damage incurred by using this material is not the sender's responsibility. The sender's entire liability will be limited to resupplying the material. Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
Re: Fw: Message Sequence Problem
Title: Message
Well
put. I would just argue that the incremental cost of programming
a reliable message sequence is usually too small to worry about. The
biggest expense is the time wasted trying to convince someone it's
important. So, if you're in a position to introduce best
practices when nobody's looking, I'd go
for it.
-Original Message-From: Wes Poteet
[mailto:[EMAIL PROTECTED] Sent: Wednesday, November 10, 2004 3:22
PMTo: [EMAIL PROTECTED]Subject: Fw: Message
Sequence ProblemThe list
of the requirements for sequential processing does exist and is the final
word. If any of the restrictions are not met then IBM does not guarantee
(assure?) that messages will be received in the same sequence as they were
sent. In general, if a network is involved, the requirements are not met
(though there are special circumstances when, even with a network, they are
met). There is anecdotal evidence out there that, when in that unguaranteed
state, the likelihood of having the sequence changed is definitely nonzero
(people have demonstrated that it does happen) but is very small. The
rule of thumb I use is that if the sequence of the messages is crucial to the
business application then the application must incorporate code to ensure the
messages are ordered appropriately. It is an appropriate topic for
discussing cost/benefit: just how critical is having one message out of
sequence out of 'n' messages? More than one design team has concluded
that a rare error, probably corrected manually, was more acceptable than the
cost and processing overhead of ensuring every message was in order.
Wes Poteet -- IBM Global Services -- WMQ
SupportIBM Certified Specialist - MQSeries; Solutions Expert - MQSeries;
Developer - MQSeriesWes Poteet/Boulder/IBM 877-513-1981 (t/l 273-1135)
Bassett, NE [[Central Time]]"Wesiderata" on BlueNet --
see w3.irc.ibm.com -- rooms #MQSeries and #WesPoteet"[EMAIL PROTECTED]"
for e-mail and SameTime "WesPoteet" on AOL IM, Yahoo IM, or MSN
IMMoral certainty is always a sign of cultural inferiority. The more
uncivilized the man, the surer he is that he knows precisely what is right and
what is wrong. All human progress, even in morals, has been the work of
men who have doubted the current moral values, not of men who have whooped
them up and tried to enforce them. The truly civilized man is always skeptical
and tolerant, in this field as in all others. His culture is based on "I am
not too sure." -- H. L.
Mencken-
Forwarded by Wes Poteet/Boulder/IBM on 2004-11-10 05:08 PM -
Rick Tsujimoto
<[EMAIL PROTECTED]> Sent by: MQSeries List
<[EMAIL PROTECTED]>
2004-11-10 03:17 PM
Please respond
toMQSeries List
To
[EMAIL PROTECTED]
cc
Subject
Re: Message Sequence
Problem
The DQM has a small list of requirements that
pertain to sequential retrieval of messages. If you followed them, you
should have no problems. I have an ftp-like application and never had an
issue with message order. Of course, there are those who advocate using
msgid/correlid, but that shouldn't be necessary as long as the requirements
stated in the DQM are met.
Wesley Shaw
<[EMAIL PROTECTED]> Sent by: MQSeries List
<[EMAIL PROTECTED]>
11/10/2004 03:37 PM
Please respond
toMQSeries List
<[EMAIL PROTECTED]>
To
[EMAIL PROTECTED]
cc
Subject
Message Sequence
Problem
The messages being processed are
not in order when they are processed fromthe MVS local queue, but
processing the same request through a local queueon the Unix box, the
messagesare processed in order.What are the rules for processing
of message in order? Seems like all thesame Priority would process
FIFO. Can we guarentee the order of the messageto be got same as they were
put ?MVS view of 416345 Local QueueAll 153 bytes except the *
record which is 9 bytes,all messages Priority 1 with
persistenceTransmit queue MSGDLVSQ(PRIORITY), Local Unix queue
MSGDLVSQ(PRIORITY)MVS Local queue Message delivery sequenc
Re: Message Sequence Problem
Message order can be preserved by following certain conditions, most which are detailed in the various programming guides. Any designs depending on those conditions, however, have potential to break if those conditions change, either within your enterprise or in a future product release. Down the road, who will remember that some application developed 5 years ago may go south if another channel is added between two qmgrs or workload increases and two programs are needed to send messages. On the otherhand, it's trivial to use logical message groups or a msgid/correlid mechanism to unconditionally preserve message order. Take the time now and save problems later. A simple and elegant method is: On put, let MQ generate unique msgid's On first message of sequence, put a special value in correlid On each message after the first, put msgid of the previous message in the correlid Send trailer message to indicate no more messages in the sequence. On first get, retrieve first message with special value in correlid On subsequent gets, use msgid from previous msg to selectively retrieve next message Continue until trailer message is encountered Regards, Dennis -Original Message- From: Wesley Shaw [mailto:[EMAIL PROTECTED] Sent: Wednesday, November 10, 2004 12:38 PM To: [EMAIL PROTECTED] Subject: Message Sequence Problem The messages being processed are not in order when they are processed from the MVS local queue, but processing the same request through a local queue on the Unix box, the messages are processed in order. What are the rules for processing of message in order? Seems like all the same Priority would process FIFO. Can we guarentee the order of the message to be got same as they were put ? MVS view of 416345 Local Queue All 153 bytes except the * record which is 9 bytes, all messages Priority 1 with persistence Transmit queue MSGDLVSQ(PRIORITY), Local Unix queue MSGDLVSQ(PRIORITY) MVS Local queue Message delivery sequence FIFO The first list is a local queue on MVS which is remote to the putting application. 00416345RECC,CMPL03CMPL 20041108115902200411081225512004 00416345RECC 03CMPL 20041108115902200411081225512004 00416345RECC 02ARRV 20041108115902200411081225512004 00416345RECC 02ENRT 20041108115902200411081225512004 00416345RECC 02DISP 2004110811590220041108122551 00416345RECC 02ASSN20041108115902 00416345* 00416345RECC 01UNAS This second view is a local queue on Unix of the same put process as above. Unix view of 416345 9 00416345RECC CMPL03CMPL ' 10 00416345RECC 03CMPL 11 00416345RECC 02ARRV 12 00416345RECC 02ENRT 13 00416345RECC 02DISP 14 00416345RECC 02ASSN 15 00416345RECC 01UNAS 16 00416345* Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
Fw: Message Sequence Problem
The list of the requirements for sequential processing does exist and is the final word. If any of the restrictions are not met then IBM does not guarantee (assure?) that messages will be received in the same sequence as they were sent. In general, if a network is involved, the requirements are not met (though there are special circumstances when, even with a network, they are met). There is anecdotal evidence out there that, when in that unguaranteed state, the likelihood of having the sequence changed is definitely nonzero (people have demonstrated that it does happen) but is very small. The rule of thumb I use is that if the sequence of the messages is crucial to the business application then the application must incorporate code to ensure the messages are ordered appropriately. It is an appropriate topic for discussing cost/benefit: just how critical is having one message out of sequence out of 'n' messages? More than one design team has concluded that a rare error, probably corrected manually, was more acceptable than the cost and processing overhead of ensuring every message was in order. Wes Poteet -- IBM Global Services -- WMQ Support IBM Certified Specialist - MQSeries; Solutions Expert - MQSeries; Developer - MQSeries Wes Poteet/Boulder/IBM 877-513-1981 (t/l 273-1135) Bassett, NE [[Central Time]] "Wesiderata" on BlueNet -- see w3.irc.ibm.com -- rooms #MQSeries and #WesPoteet "[EMAIL PROTECTED]" for e-mail and SameTime "WesPoteet" on AOL IM, Yahoo IM, or MSN IM Moral certainty is always a sign of cultural inferiority. The more uncivilized the man, the surer he is that he knows precisely what is right and what is wrong. All human progress, even in morals, has been the work of men who have doubted the current moral values, not of men who have whooped them up and tried to enforce them. The truly civilized man is always skeptical and tolerant, in this field as in all others. His culture is based on "I am not too sure." -- H. L. Mencken - Forwarded by Wes Poteet/Boulder/IBM on 2004-11-10 05:08 PM - Rick Tsujimoto <[EMAIL PROTECTED]> Sent by: MQSeries List <[EMAIL PROTECTED]> 2004-11-10 03:17 PM Please respond to MQSeries List To [EMAIL PROTECTED] cc Subject Re: Message Sequence Problem The DQM has a small list of requirements that pertain to sequential retrieval of messages. If you followed them, you should have no problems. I have an ftp-like application and never had an issue with message order. Of course, there are those who advocate using msgid/correlid, but that shouldn't be necessary as long as the requirements stated in the DQM are met. Wesley Shaw <[EMAIL PROTECTED]> Sent by: MQSeries List <[EMAIL PROTECTED]> 11/10/2004 03:37 PM Please respond to MQSeries List <[EMAIL PROTECTED]> To [EMAIL PROTECTED] cc Subject Message Sequence Problem The messages being processed are not in order when they are processed from the MVS local queue, but processing the same request through a local queue on the Unix box, the messages are processed in order. What are the rules for processing of message in order? Seems like all the same Priority would process FIFO. Can we guarentee the order of the message to be got same as they were put ? MVS view of 416345 Local Queue All 153 bytes except the * record which is 9 bytes, all messages Priority 1 with persistence Transmit queue MSGDLVSQ(PRIORITY), Local Unix queue MSGDLVSQ(PRIORITY) MVS Local queue Message delivery sequence FIFO The first list is a local queue on MVS which is remote to the putting application. 00416345RECC,CMPL03CMPL 20041108115902200411081225512004 00416345RECC 03CMPL 20041108115902200411081225512004 00416345RECC 02ARRV 20041108115902200411081225512004 00416345RECC 02ENRT 20041108115902200411081225512004 00416345RECC 02DISP 2004110811590220041108122551 00416345RECC 02ASSN 20041108115902 00416345* 00416345RECC 01UNAS This second view is a local queue on Unix of the same put process as above. Unix view of 416345 9 00416345RECC CMPL03CMPL ' 10 00416345RECC 03CMPL 11 00416345RECC 02ARRV 12 00416345RECC 02ENRT 13 00416345RECC 02DISP 14 00416345RECC 02ASSN 15 00416345RECC 01UNAS 16 00416345* Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
Re: Message Sequence Problem
The DQM has a small list of requirements that pertain to sequential retrieval of messages. If you followed them, you should have no problems. I have an ftp-like application and never had an issue with message order. Of course, there are those who advocate using msgid/correlid, but that shouldn't be necessary as long as the requirements stated in the DQM are met. Wesley Shaw <[EMAIL PROTECTED]> Sent by: MQSeries List <[EMAIL PROTECTED]> 11/10/2004 03:37 PM Please respond to MQSeries List <[EMAIL PROTECTED]> To [EMAIL PROTECTED] cc Subject Message Sequence Problem The messages being processed are not in order when they are processed from the MVS local queue, but processing the same request through a local queue on the Unix box, the messages are processed in order. What are the rules for processing of message in order? Seems like all the same Priority would process FIFO. Can we guarentee the order of the message to be got same as they were put ? MVS view of 416345 Local Queue All 153 bytes except the * record which is 9 bytes, all messages Priority 1 with persistence Transmit queue MSGDLVSQ(PRIORITY), Local Unix queue MSGDLVSQ(PRIORITY) MVS Local queue Message delivery sequence FIFO The first list is a local queue on MVS which is remote to the putting application. 00416345RECC,CMPL03CMPL 20041108115902200411081225512004 00416345RECC 03CMPL 20041108115902200411081225512004 00416345RECC 02ARRV 20041108115902200411081225512004 00416345RECC 02ENRT 20041108115902200411081225512004 00416345RECC 02DISP 2004110811590220041108122551 00416345RECC 02ASSN 20041108115902 00416345* 00416345RECC 01UNAS This second view is a local queue on Unix of the same put process as above. Unix view of 416345 9 00416345RECC CMPL03CMPL ' 10 00416345RECC 03CMPL 11 00416345RECC 02ARRV 12 00416345RECC 02ENRT 13 00416345RECC 02DISP 14 00416345RECC 02ASSN 15 00416345RECC 01UNAS 16 00416345* Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
Message Sequence Problem
The messages being processed are not in order when they are processed from the MVS local queue, but processing the same request through a local queue on the Unix box, the messages are processed in order. What are the rules for processing of message in order? Seems like all the same Priority would process FIFO. Can we guarentee the order of the message to be got same as they were put ? MVS view of 416345 Local Queue All 153 bytes except the * record which is 9 bytes, all messages Priority 1 with persistence Transmit queue MSGDLVSQ(PRIORITY), Local Unix queue MSGDLVSQ(PRIORITY) MVS Local queue Message delivery sequence FIFO The first list is a local queue on MVS which is remote to the putting application. 00416345RECC,CMPL03CMPL 20041108115902200411081225512004 00416345RECC 03CMPL 20041108115902200411081225512004 00416345RECC 02ARRV 20041108115902200411081225512004 00416345RECC 02ENRT 20041108115902200411081225512004 00416345RECC 02DISP2004110811590220041108122551 00416345RECC 02ASSN20041108115902 00416345* 00416345RECC 01UNAS This second view is a local queue on Unix of the same put process as above. Unix view of 416345 9 00416345RECC CMPL03CMPL ' 10 00416345RECC 03CMPL 11 00416345RECC 02ARRV 12 00416345RECC 02ENRT 13 00416345RECC 02DISP 14 00416345RECC 02ASSN 15 00416345RECC 01UNAS 16 00416345* Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
