On 18/04/13 17:51, Gordon Sim wrote:
I also agree it is important to be able to control what happens when a
(slow or disconnected) subscriber starts building up a backlog of
messages. In many cases I think this is something that should really
be a policy on the exchange or 'topic' (i.e. how much backlog are any
of the subscribers allowed to build and what happens when they exceed
that).
Hmm, I'm not sure. I guess that it depends exactly what you mean by a
policy on the exchange. I've seen some of the work that has been going
on on the "rebind" thread and being able to redirect messages to an
alternate consume when queue threshold exceed triggers would be useful
to me in some scenarios so I'm keen to loot at that, but the current
circular queue model seems to fit my scenario the best in most
circumstances. Having the exchange care about how much backlog
subscribers are allowed suggests in my mind some sort of discard when
that limit is reached. For sure there are cases where I'd be happy with
that, and protecting producers and/or the well behaving consumers is
high on my list of things I like. The nice thing about circular queues
however is that the most recent item always hits the queue and it's the
oldest items that get blatted if the queue depth grows. That's something
I really need!! Indeed If I had a "wish list" what I'd *really* like is
a fixed depth stack, in other words a lifo rather than a fifo. As I
mentioned previously I've got a hight rate, low latency system and
perhaps I should have mentioned that the most recent items are
"generally" the most interesting, certainly if I'm put in a position of
dumping items it's the oldest ones I'd kill first. With a circular queue
clearly I get that behaviour, but as the queue fills the latency gets
impacted somewhat.
There may be games with priority queues and the like that could make
this better (and to be fair it has never been something that has
critically bitten us) but it seems like overkill, I'd have assumed that
things like priority queues and messing around with item TTL all use
cycles. TBH I'm possibly in a slightly weird case of preferring things
to be "lean 'n' mean".
It may also be important to allow some per-subscription control.
I think having options for finer grained control would be great. I'm all
for sensible defaults, but as I mentioned previously the queue defaults
have bitten me hard and I had to resort to sub-optimal approaches such
as queue routes and similarly the lack of control of things like
credit/capacity on bridges almost drove me over the edge last year (glad
that one has been fixed with the "credit control") it's still be nice to
be able to configure bridges in exactly the same way as any client
though, I've never quite understood why it wasn't possible. I *nearly*
had to resort to building and external bridge application because of the
lack of control, which would have been a bit tragic :-)
The intention for selectors when used with an exchange is to filter
*before* enqueueing, i.e. to further restrict the set of messages the
subscriber sees without having those build up in some broker side temp
queue. That isn't yet done, but I have a JIRA open for it.
That sounds absolutely excellent. I really like message selectors and
this sounds like a really nice approach.
[Specifically with respect to the headers exchange, I'm not sure it
offers much benefit over selectors (once working as expected). I would
need to do some benchmarking to be sure, but the matching logic in the
headers exchange is not particularly clever and my guess would be
there would be little difference in performance as compared with
evaluation of an equivalent selector.]
So to be honest my issue is I suspect more subtle than just "pure
technical". When I started architecting my system I'd have given my eye
teeth and quite a few other body parts to have been able to use message
selectors. I'd used them loads and frankly the boolean capabilities
would have stopped me from losing the last vestige of sanity I once had
:-/ However they didn't exist and that's partly my slight dilemma. I
have a non-trivial number of consumers and therefore a potential
non-trivial integration issue. I feel pretty heartened that being able
to apply selectors between exchange and queue is ultimately good news
but I'd be "very sad" if I had to change all subscriptions in lock-step.
If it's possible to have a mixed economy whereby I can have some
consumers using AMQP 0.10 and current headers bindings and some using
AMQP 1.0 and selectors then I can have a controlled migration. In an
ideal world I'd like to be able to pump controlled data into my
reference system and have a consumer with headers bindings and a
consumer with the equivalent selector so that I could prove any binding
changes systematically too.
I guess that the point is that I've got a current operational system so
having to change any of the key characteristics introduces mission risk
so I'd be uncomfortable without options to mitigate this. Clearly if the
"to be" offers increased flexibility and control then that would really
help.
I hope that this makes sense. Please don't take any of this as being
intentionally negative, I'm aware of the need to evolve, but OTOH I've
got a mission critical Qpid based system that I'm keen to protect.
Re "but the matching logic in the headers exchange is not particularly
clever" yeah, I'd gathered that. I've not looked in detail but it looked
like it was doing linear searches. It'd be nice to see some benchmarks
comparing headers matches with equivalent selectors.
heh heh <evil>given that there's almost certainly a selector equivalent
to a headers match it aught to be possible to take an existing headers
binding and auto translate it - fancy giving that a go as a configurable
option on qpid::messaging</evil> :-D
As an aside any idea if thought has gone into selectors with respect to
binary Vs UTF8 strings. You may recall some interoperability comedy in
the past whereby C++ (qpid::messaging at any rate) tended to default to
setting headers as binary strings whereas Java would use UTF8. I'd be
*very keen* to know if thought has gone in to string interoperability,
it's very easy to miss without testing in a heterogeneous environment.
I suppose all of this is also suggesting that the XML exchange is likely
to be on its way out. I've never really used it myself but I expect
people who do will be faced with similar questions.
The selector syntax the c++ broker supports (in 0.22) is as defined in
the registered extension. Though there may be one or two gaps still
to be implemented (e.g. access to all the AMQP 1.0 defined fields) it
does not deviate into non-standard syntax in any way. The c++ client
should be able to use selectors against the java broker over 1.0 (I
haven't yet got around to testing that however), or indeed anything
else that supports that selector syntax (e.g. SwiftMQ).
I guess that I don't know enough about the syntax yet. I was thinking
primarily around JMS Message Selectors. It'll be somewhat unfortunate
and I suspect open up fair scope for confusion if AMQP Message Selectors
have radically different syntax and capabilities compared with JMS
Message Selectors. TBH it would be a "right pain". Ultimately the JMS
client would have to support JMS Message Selectors to be JMS compliant
so there may be an implicit mapping to AMQP Message Selectors if they
happen to be different. I don't know enough about the detail but it's
making me a little nervous, perhaps Rob knows?
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
