Hi Gordon,
Comments inline below.
Frase
On 09/04/13 19:07, Gordon Sim wrote:
On 6 April 2013 10:40, Fraser Adams <[email protected]> wrote:
"
The x-bindings property is not currently supported for AMQP 1.0 in
nodes or links. This has really been a question of priorities rather
than ruling out any mapping. Though I don't think a generic binding
solution is appropriate for the 1.0 implementation, I'm eager to here
of use cases that would be affected here and see how best to address
them.
"
I use the headers exchange almost
exclusively in my operational system and my Addresses are of the form:
string address = "testqueue; {create: receiver, node: {x-declare:
{arguments: {'qpid.policy_type': ring, 'qpid.max_size': 500000000}},
x-bindings: [{exchange: 'amq.match', queue: 'testqueue', key: 'data1',
arguments: {x-match: all, data-service: amqp-delivery, item-owner:
fadams}}]}}";
Does each consumer create its own queue?
Sort of. Each "logical" consumer does, but a logical consumer may
comprise several physical nodes, they all use the same queue which
allows distribution of messages across the physical nodes giving the
ability to scale out quite easily.
I actually think this will be both
simpler and more powerful over 1.0 as the plan is to allow you to subscribe
to an exchange (which you can already do) using a selector.
"Subscribing to an exchange" is a term that makes me shudder from past
experience.
I'll try to explain, my system is perhaps a little unusual. I've got a
pretty large federated topology with lots of *real time* C++ producers
at one end of the topology and something of a mixed economy of mainly
Java and C++ (and a few Python) consumers mostly at the "right hand
side" having passed through some intermediate "data consolidating"
brokers. I've also got a few higher rate consumers subscribing directly
off the intermediate brokers. The mean message size tends to be
reasonably large, something like 50K but the tails are long and some
messages might be tens of MB.
Hopefully you noticed what I had in the x-declare of the example
Address. I'm using circular queues for a reason - high rate real time
producer, and we generally actually use 2GB queues to give moderate
elastic buffering given data rates and message size.
I've got nothing against the logical "subscribing to an exchange"
pattern and it is pretty common to see addresses simply expressing the
topic, but as you are aware "under the hood" this creates an autodelete,
exclusive, reject policy, default size queue!! I can't live with any of
those characteristics let alone all of them :-)
Indeed when I said it makes me shudder I'm actually thinking of our past
federation experiences, as you know static and dynamic routes are more
explicitly described as static and dynamic /exchange/ routes, because
they logically allow exchanges to be connected together. This is
conceptually very much like one exchange subscribing to another
exchange, but again they transparently create a queue almost completely
unsuitable for my purpose :-( We've had to use queue routes simply
because I can control the queue config. You wouldn't believe how fast
default size reject queues cause links to "blow" if one of my consumers
misbehaves, which clearly results in total data loss to all my consumers
not just the naughty one - not ideal.
I'd have *really* liked to be able to combine the rather elegant dynamic
binding propagation of dynamic routes with the ability to specify the
configuration of the queue/buffering used by the link with a great deal
more precision than the defaults.
Hopefully that background give some insight into my "subscribing to an
exchange" nerves :-/
Beyond that again if you notice from the example Address I use the
headers exchange pretty much universally. To be fair my architecture is
pretty much a publish/subscribe architecture and in the early days I
started with topics, but to be honest the largely static "data model"
and fairly inflexible wild-carding caused us to move to the headers
exchange pretty quickly. It's a lot easier to add new "data types" for
subscribing to with the headers exchange.
It hadn't escaped my notice that the bindings stanza was "x-bindings"
but as far as I've been able to work out that's the only way to create
bindings between the headers exchange and a named queue, indeed I
thought that x-bindings was the only way to create bindings between say
a user created topic exchange and a queue, though I may be wrong there.
If you can suggest how to say subscribe to a headers exchange without
using x-bindings that'd be neat, it'll be something I've totally missed.
Re "using a selector " I have to say that I'm definitely a big fan of
message selectors. It's worth pointing a few things out though. I
suspect that had message selectors been available when I started out
with my system I might well have used the topic exchange (with a named
queue!) plus message selectors. I've used JMS a lot in the past and
that's at pattern that worked well and was very flexible, however as I
said earlier the topic exchange on its own didn't cut the mustard for my
needs so my producers describe their data purely by means of
properties/headers.
I guess you know this already but message selectors tend to behave as
filters, so as I understand it messages would pass from exchange to
queue and the selector would apply a filter to what gets passed off the
queue to a consumer. I guess that to my way of thinking selectors
provide most benefit in the realm of "fine grained" filtering in that
they allow quite sophisticated boolean expressions to be applied.
Now in my scenario quite clearly message selectors will be a boon and
offer some really useful fine grained control (I've wanted to be able to
do "not" for ages - you might just about recall one of my very early
posts on the headers exchange asking about a not operator). However I
think that message selectors are best applied *in addition to* a
subscription, without that loads of messages get delivered onto the
queue which the consumer isn't interested in, which a) fills up the
queue unnecessarily and b) overworks the message selector logic (which I
suspect is more expensive than the exchange selection logic). In my case
in particular producers deliver quite a diverse range of data at very
high rate so filling up consumer queues unnecessarily would be pretty
bad news.
So in précis while I really like message selectors I'd want them *in
addition to* headers bindings and good granularity of queue control (I
don't ask for much).
Another point about selectors. As I say I'm excited about them, but they
are a new feature in the C++ broker (should be in 0.22 is that
correct?). Andrew has done some great work so far, but he has very much
said that the initial priority is on features and not interoperability.
I'm fine with that initially, but you will undoubtedly have gathered
from many of my posts that I'm really keen on maximum interoperability
and cohesion. I'd really like to see the C++ message selectors
ultimately become completely interoperable with JMS Message selectors.
I've got no problem with an "x-selector" heading in the Address string
but I'm really meaning at an API level too. I'd like to see JMS message
selectors from JMS Clients propagate to the C++ and Java brokers and I'd
like to see qpid::messaging extended to allow a JMS style message
selector string to by passed to the createReceiver() call on Session.
Clearly it's early days for C++ broker message selectors, but if that
strategy wasn't part of the roadmap I'd be pretty uncomfortable.
The key then
(if I understand the usecase) would be to ensure that the subscrption had
the right lifecycle (i.e. you could reconnect to it without loss of
messages while disconnected),
Well the lifecycle aspects are certainly part of it, hopefully my
comments above have explained the other aspects of the set of use cases
that are fairly critical to me.
One thing that's not clear from your write up is whether the x-bindings
property you are talking about is in the messaging client, or whether it is
the underlying x-bindings property that gets passed to the broker. I'm
guessing the latter which would also affect AddressStrings created by JMS
consumers and via QMF? (Although I use the C++ broker most of my consumers
use JMS).
It is in the messaging client. The x-bindings (and x-declare, x-subscribe),
map directly to 0-10 commands. There is no analogous mechanism for
*generic* binding operations in 1.0 - i.e. the ability to bind almost
anything to anything, regardless of the node you are sending to or
receiving from. However when you 'subscribe' to an exchange node, the
broker can infer bindings from the link attachment, generally through the
specification of a filter. In your case it sounds like that might be the
applicable pattern over 1.0 and given the matching is based on examination
of headers it could even be a selector.
I guess I have to admit that I don't understand what you are saying
entirely, to be honest one common theme that seems to be building in my
mind is that AMQP 1.0 seems somewhat to be a bit of a retrograde step
over AMQP 0.10. I'm sure my current lack of understanding is making my
initial assumption a tad unfair but would you be able to summarise the
"compelling features" that AMQP 1.0 offers over AMQP 0.10 (aside from
AMQP 1.0 having been ratified by OASIS - that's probably a good thing,
but how much "design by committee" goes on at OASIS? is it closer to
IETF or closer to ITU-T?)
So knee jerk comments aside :-) Re "It is in the messaging client. "
does that statement imply then that it remains possible to create the
queues and bindings to the headers exchange via QMF "create" calls to
the Broker? If so then that's something positive, but I'm then confused
by the statement "There is no analogous mechanism for *generic* binding
operations in 1.0 - i.e. the ability to bind almost anything to
anything, regardless of the node you are sending to or receiving from ",
if it is possible to do this by QMF "create" surely that remains a
mechanism for generic binding. Sorry I'm sure I'm being dumb, I really
need to find the time to read up on the AMQP 1.0 spec.
I guess in simple terms what I'm asking is how on earth do you set up a
subscription to a headers exchange in AMQP 1.0 along with the ability to
control the "buffering" between producer and consumer - pretty much what
my Address example above is doing.
One thing I should make clear is that 0-10 is not going away any time soon.
The intention is not to force anyone onto 1.0, but simply to make it
available (and indeed I hope eventually the default for new developments).
Yeah I realise that Gordon, perhaps *panic* is too strong a term for my
feelings :-) There are clearly issues though, now that AMQP 1.0 has been
ratified as you say it does certainly seem to be something we ultimately
need to migrate to even if not immediately. I've mentioned to you and
Rob before that I'd really like to see some published roadmaps for Qpid.
So "not going away any time soon" is one thing but a statement that we
intend to support AMQP 0.10 indefinitely, or until December 2015 (random
date!!) or whatever would give people confidence or conversely an
opportunity to plan their own architectural changes well in advance.
I think one good thing about AMQP 1.0 being a ratified standard is that
it opens up opportunities for integration. A couple of our consumers
actually use ActiveMQ internally (and currently use a Camel based bridge
IIRC), it'd be much more elegant to federate directly from qpidd to
ActiveMQ - I'm assuming that having a ratified standard would make that
sort of thing more of a possibility ultimately.
As an aside Qpid remains at a 0.x version. I guess that I had sort of
assumed that there would be an intention to uplift to Qpid 1.x when AMQP
1.0 support had stabilised and was proved to be as reliable as the AMQP
0.10 support, but TBH I've not seen any traffic at all relating to the
version? Any thoughts? I have to be honest I'd bet that the 0.x version
structure puts off a whole bunch of people who would really benefit from
using Qpid. To get on my "branding" hobby horse again we need to
consider just quite how conservative many large Enterprises actually
are, it's hard enough to get some of them to consider Open Source, but
version 0.x to many implies "unreliable and unstable" which Qpid is
anything but. I think we're doing ourselves a great dis-service if we
don't put some consideration into what it would take to become version 1.0
The x- options were always going to be quite 0-10 specific. In some places
they were unavoidable.
Yeah I realise that, but the headers binding was a standard exchange in
AMQP 0.10 (don't know about AMQP 1.0). To be fair I guess the whole
Address String is Qpid specific :-)
I'm hoping that for 1.0, most cases can be met much
more simply and effectively. But again, I wouldn't want anyone to be
alarmed by my mail as 0-10 is unaffected and will continue to be fully
supported.
Thanks again, I'm not *too* alarmed :-) I guess it's good in that it
prompted me to flag up one of my most critical use cases. I hope that
the additional explanation that I've added to this mail helps clarify
things even more.
Funnily enough I was just recently in an off-list conversation with Rob
and Robbie on the subject of Ring Queues, cause I couldn't fathom why
the Java broker didn't support circular queuing :-) I suppose that the
moral is not to presuppose peoples critical use cases. I personally
couldn't give a damn about transactions for example (though I can
*really* understand why some cases need them :-) ) from my perspective
I'm using Qpid as a very high throughput (rate/message size and volume)
low latency data delivery fabric and have real time producers and
multiple consumers who care about wildly different data types. I
*really* don't want any dodgy consumer interfering with data delivery to
any of the other consumers and I *really really* don't want any
consumers interfering with my producers (I've not figured out a way of
slowing down time yet :-)) so basically I need a very decoupled pub/sub
system.
Also, I do want to get feedback exactly like this. Highlighting things that
concerns existing users, especially if they would like to move to 1.0.
Cool, hope this slightly lengthy tome is useful too.
I
can't guarantee 100% fidelity for all 0-10 address string options (some
simply cannot be sensibly mapped and would in any case tie you to qpidd so
tightly that there would be little benefit to using 1.0!). I do however
want to do what I can to ease the transition. So if I need to do some more
thinking on x-bindings, I'm happy to do that. Thanks for he feedback, its
very helpful!
