Paul, All:
Please beleive me that my preference would be not to overheat this already pretty hot
thread and just shut up and not to swim against the current. However I feel all those
arguments basically calling to ignore the official Q guarantees of the message order
under some specific (tangible and relatively easy verifiable) conditions can make a
bad advice to a fresh new designer convincing him or her it is always the right and
the best choice to follow the mentioned design recommendations or it is always
possible to do so. I will try to summarize all those arguments as brief as I can and
give a counter-arguments (I understand the arguments were perfectly legal for many
cases, nevertheless I allow myself not to consider their 'pros' because it was already
done so many times).
1) "DO NOT build a dependancy of message order into you application". Actually I am
100% agree -- but!
a) if it is possible at all and
b) if it does not create significant disadvantages for the user's business.
Counterexample: Clearing House feed and similar. User can try to get into the
arbitrary settlement at any moment of time and the processing engine must determine
its validity based on the user account state. The result of the is the sequence of
transaction confirmations or denials (let's call them records) with each record
containing a transaction id, a boolean (confirmed/denied), and a timestamp. Please pay
attention to the last one! The timestamp records the moment of time when the
confirmation or denial user was issued! It is a significant part of the result,
because, if it takes user too long to wait for the clearance of his or her deal, he
can select another clearing house -- see 1.b) above. I think it is obvious for
everyone that here we have an inherent order dependancy in the business logic.
2) "Use message segmentation or grouping". -- Back to the example above, it is
apparent that there is no natural notion of "groups" in the example. We can try to
introduce an artificial group such as "business day" etc. but it will adversely affect
the requirement 1.b) above. User experience will be seriously affected, the clearing
house won't be real-time anymore. What may be worse, the non-timestamp component of
the result can change, namely a set of confirmed and denied deals in the business day.
Possibly even more deals might be confirmed than in real-time engine, but they can
easily be *different* deals and, because of that, we may expect, e.g. some
legislations directly prohibiting such artificial grouping practice (I am sure some of
you have met such conflicts in your practice).
3) "The point is not having a DLQ is generally bad for everybody. A simple application
mistake could bring your channel down and therefore bring
everyone down". -- We discussed this several time. My points are:
a) The "simple application mistake" of this sort (usually it is passing through too
long message or not having a proper handshakes on data streams) means that the
application is not ready to production. The dot. Data must be checked and validated
before hitting the transport. In case the applications are not at the same "trust
domain" use different queues and channels for them (or even queue managers -- it is
not necessary but sometimes gives additional advantages when users want to minimize
the possible adverse effect of one application to another).
b) If a "simple application mistake" does occur, it is impossible to detect it ASAP
and not to continue the processing (e.g. in financial organization continuing the
processing after the "simple mistake" may expose it to serious financial and legal
risks). Shutting down the channel can be not a bad choice to fire the drill. Using DLQ
makes investigations much more difficult (even with all proper handlers in place) and
some not-evident error can even sneak in unnoticed.
4) "..Use sequence number..". The sentence was never stated in enough details, so let
me make it full before objecting to it. "You can use sequence number (together with
some home-grown unique publisher identification if multiple publishers are allowed on
the queue) and fully restore the message order on the application level if it is so
precious for you". -- I agree, this is possible. TCP stack does it, at the first
place! My point is that although this all is doable, it is not free from the following
points of view:
a) It is not easy to write a really bullet-proof order restoring layer. Anyone who
thinks it is not so is welcome to look into some good existing code (take the source
for Linux TCP stack e.g.). As such it does not make sense to re-invent this not really
simple general purpose wheel for each and every applications. Why IBM does not provide
a re-usable code example? I dare to speculate -- because the real thing would too
bulky and complicated to serve for the novice instruction purposes. But isn't giving
the recommendation which is difficult to follow without the working example even worse?
b) It is difficulet to implement this feature usable in practice without configuration
parameters (like time window, assembly pipeline length, maxumum buffer sizes etc).
Again, see the TCP configuration parameters for a good example of what can be there.
Application user would have to set the values for this parameters and they can
interfere in a different unexpected way with TCP system wide parameters, badly
degrading the performance (I had a bad experience of this kind with one my socket
application which was simpler than MQ channel).
c) It is like destroying some invariant (message order) you can often (not always) get
"for free" (satisfy the conditions which are not actually too restrictive). It creates
a significant (really) perfomance penalty. I remember someone (Bobbee, I beleive)
saying that several (order of 10) messages per second processed by the MQSeries on the
mainframe is a normal performance. Maybe the home-grown message order restoring
engines can be kept accountable, at least partially for such a low performance?
d) As a consequence of a) I can say: neglecting gurantees that MQ provides wrt
preserving the message order leads to the more complex solution. So to me anyone who
has to make a decision whether to use DLQ and restore message order in the
applications or try to use MQ "embedded" order guarantee should be have a really good
reason to select the 1st choice and select the 2nd choice by default, not the other
way around.
Hopefully this memo will be useful for someone and not offensive. I apologize in
advance for any potential misunderstanding.
Pavel
Paul Clarke
<paulg_clarke@UK. To: [EMAIL PROTECTED]
IBM.COM> cc:
Sent by: MQSeries Subject: Re: retrieve order
List
<MQSERIES@AKH-Wie
n.AC.AT>
09/19/2002 09:34
AM
Please respond to
MQSeries List
>Hello All,
>My understanding is that TCP guarantees the order of bytes. For all and
every message and between messsages. As soon as a single >channel does not
open more than one simultaneous connections between same nodes -- which it
cannot do, if I am not mistaken.
>My understanding is that the "different routes" mentioned in documentation
are the different paths consisting of MQSeries >channels. For example, in a
cluster, you can easily end up with more than one possible path from one
Queue Manager to another. >Or, you can create more than one SENDER-RECEIVER
channel manually. In this case, when messages can select different
channels, >FIFO will be broken. The DLQ is another possible problem (that's
why I always advice against it, a opposite to the IBM's official
>recommandations).
>It would be nice if Paul or someone else from IBM prove or disprove that.
Pavel,
I think I already explained this in my previous append on this subject but
let me confirm again.
1/ TCP/IP guarantees byte ordering so it doesn't matter what path the IP
packets take.
2/ In the rules; when IBM talks about multiple paths it is talking about
multiple MQ paths ie. multiple channels.
3/ I do agree that you can 'easily' end up with more than one path in an MQ
cluster but it shouldn't happen by accident - you would have to deine
multiple clusrcvrs or multiple aliases or multiple gateways.
4/ I agree the DLQ is a source for out of sequence, however, this should be
a rare occurence ie. queue full
Your point about DLQ is well taken but if people followed all of our advice
we wouldn't be having this conversation at all.
We advise that you DO NOT build a dependancy of message order into you
application in the first place.
It is very easy to write your code so that if you do need multiple messages
in a single 'business' message that the receiving application can deal with
them. Common approaches are :-
1/ Use message segmentation or grouping
2/ Use two queues - one for the primary message and one for all the
secondary pieces - use something like correl id to be a sequence number so
you can retrieve the messages off the secondary queue in the order you want
3/ Use a single queue but always do a GET(correlid=1) first. That message
tells you whether there are correlid=2,3,4 etc etc that you should now go
get.
The point is not having a DLQ is generally bad for everybody. A simple
application mistake could bring your channel down and therefore bring
everyone down. If you have a DLQ then all other arguments are pointless -
your applications should be able to deal with out of order messages.
Hope this helps,
P.
Paul G Clarke
WebSphere MQ Development
IBM Hursley
Instructions for managing your mailing list subscription are provided in
the Listserv General Users Guide available at http://www.lsoft.com
Archive: http://vm.akh-wien.ac.at/MQSeries.archive
--
This e-mail may contain confidential and/or privileged information. If you are not the
intended recipient (or have received this e-mail in error) please notify the sender
immediately and destroy this e-mail. Any unauthorized copying, disclosure or
distribution of the material in this e-mail is strictly forbidden.
Instructions for managing your mailing list subscription are provided in
the Listserv General Users Guide available at http://www.lsoft.com
Archive: http://vm.akh-wien.ac.at/MQSeries.archive
