I still think you are looking at it the wrong way Clive.
so consider your scenario
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
now think about it a different way and imagine adding the queue and the
bindings via qpid-config and receiving via say:
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}}}"
Rxer 2 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}}}"
Rxer 3 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}}}"
In that case I doubt you'd have any issue assuming that the receivers
would likely alternately pick the next message off the queue.
The reality is that with the receivers that you've specified that's
*exactly* what you've done.
In AMQP terms you are addressing the queue node named queue1 from all
three receivers.
I'm not *quite* sure why your customers have an aversion to queues,
though I guess that the take up memory so perhaps that's why?
I've not thought too hard about this, but I wonder if there's a way to
achieve something like you seem to be trying to achieve via a queue
browser - possibly not in AMQP 0.10 but perhaps in AMQP 1.0 with message
selectors. TBH I haven't played too much with browsers.
I *think* what you seem to be expecting is that the filter is associated
with the receiver, but it's not. In AMQP 0.10 bindings are formed
between exchanges and queues, so there's no way to specify an
association between a particular binding and a given receiver without
also having a queue associated with the receiver - unless you apply your
filtering on the client side.
As I said in my earlier response I think that the correct way to achieve
what amounts to client side filtering would be to expose some of the
logic of qpidd such that it could be used as building blocks for a more
general AMQP 1.0 container then you could do a link attach from your
actual receivers to the client side node in your client container, but
TBH none of that has been considered yet, but I'm not convinced that
what you are suggesting actually maps to either AMQP 0.10 or AMQP 1.0
Re "to support the behavior that, it would appear, most developers would
expect to see" so I'm not convinced that it really is what most
developers would expect if they really thought about it. If you replay
the question to them in the way I suggested above where you add the
queue and bindings via qpid-config I expect that your straw poll would
have a different outcome.
Frase
On 11/02/14 19:34, CLIVE wrote:
Ted,
Thanks for the response and your comments.
I have had to handle the case of multiple Receivers attaching to the
same queue on several occasions; primarily because the customer has
felt that it was easier to handle one queue with multiple bindings (up
to 100), rather than having a hundred queues with single bindings;
message order was also a contributing factor.
The point of the post was just to raise it as a possible issue for
future improvement..
I carried out a straw poll of 10 developers today at work. I gave them
the two examples previously described and asked what they would expect
to happen for the case where multiple Receivers were created for the
same queue. They all expected the correct Receiver to be returned from
the nextReceiver method, not the undeterministic behaviour that they
would see.
I wouldn't have thought that it would take that much code/effort to
add some additional functionality in the messaging API Implementation
to support the behavior that, it would appear, most developers would
expect to see. If I find some time I will take a look and see how it
could be done.
Clive
On 10/02/2014 22:17, Ted Ross wrote:
Clive,
What you are observing is what I expect: In the second scenario
where you use the same queue for each of the three receivers, the
receiver that receives any particular message will be non-deterministic.
This is because the binding key is applied between the exchange and
the queue (i.e. it is used to determine which queue(s) the message
should be enqueued on). Multiple receivers on a queue will receive
messages from the queue in an undetermined order, but no message
shall be delivered to more than one receiver. In the second case,
all of the messages are placed on the same queue in the order in
which they arrive. The queue acts as a buffer between the routing
rule that matched the message and the receiver that provided the
routing rule.
It would be simpler to do the following:
Rxer 1 - "amq.topic/bill; {link: {x-declare: {auto-delete:true}}}"
Rxer 2 - "amq.topic/ben; {link: {x-declare: {auto-delete:true}}}"
Rxer 3 - "amq.topic/tim; {link: {x-declare: {auto-delete:true}}}"
This will give you the determinism you want. This will cause the
creation of a temporary queue for each receiver that will receive the
messages that match the topic key (following the slash in the address).
-Ted
On 02/10/2014 04:39 PM, CLIVE wrote:
Fraser,
Thanks for the response. The real problem is that the behavior of a
Receiver is different depending on the multiplicity of the binding
strategy used. If you use a single queue with a single binding then
messages will get delivered to the required receiver. If you use
multiple Receivers bound to the same queue, the Receiver called by
the messaging API, when delivering a message to your application,
may not be the one that you think!!
So if I create three Receivers in the same application, with the
following bindings (note unique queue names)
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue2; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue3; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And then send a message on the amq.topic exchange with a subject of
'tim'. Then Rxer3 will get returned by the 'nextReceiver' method on
the associated Session object.
But if I change the bindings so they related to the same queue
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And send the same message again, Which Receiver would you expect to
get returned from the sessions nextReceiver method?
I would expect the same result as in the first example, Rxer 3. But
this does not happen, anyone of the three receivers might get called.
This doesn't seem right to me and as a result you have to produce
quite a bit of application level logic to handle this scenario;
especially when your bindings are being passed down to you
dynamically by several client applications.
Hope this explains it a bit better than my last attempt.
Clive
On 07/02/2014 10:03, Fraser Adams wrote:
On 06/02/14 19:07, CLIVE wrote:
Hi,
[snip]
The first use case requires the dynamic creation of Receivers, but
before creating a new receiver, I would like to know if I already
have a receiver that would match the required binding. This is not
possible at the moment because the binding matching algorithms are
hidden from public view; they are buried deep inside the Brokers
Exchange Implementation code.
You know that you can get the binding information from QMF don't
you Clive? I guess I'm missing what you're looking for if it's
something different than that. And I guess to be fair to get the
information via QMF you'd need a bit of code, but I'd have thought
that this would be the most appropriate way to get the information.
Out of curiosity why do you need to know if you already have a
receiver that would match the binding?
One thing that's worth mentioning, I'm suspecting that (like me)
you've mainly been using AMQP 0.10 - If I'm reading you correctly
you sound like you are dynamically creating queue nodes and passing
x-bindings.
I've been doing that for a few years, but a few weeks back I
started looking at AMQP 1.0 and that primarily takes a perspective
of addressing the topic like exchanges and the queues end up being
subscription queues and all of the stuff that relates to binding
and the like ends up in the link (not node) config.
For me at any rate that was quite a different perspective on things
(I wrote up what I was up to in the "A write up of some AMQP 1.0
Experiments" post) previously I've been focussing on the queues, so
I was dynamically creating queue nodes and passing x-bindings in
AMQP 0.10, but in AMQP 1.0 I've been addressing the exchanges
(topic type nodes) and using the link to specify what I need. For
me it took a bit of getting used to because I was so ingrained
doing it the other way, but I think I'm getting it now.
The second use case in question requires a client application to
dynamically create multiple receivers for the same queue, but with
slightly different binding keys bound to an exchange. When a
message from an exchange gets put in the queue and delivered to
the client (via a receiver)
I'm not sure if I'm correctly interpreting what you are saying
here, so you want a client that has a single queue, but each
receiver adds different binding keys right? You do know that this
will result in what amounts to an OR condition - both keys will be
bound and a message will be put on the queue if either match so
consumer A of the queue would receive messages due to consumer B's
key - is that what you mean.
The following AMQP 1.0 consumers will do what you seem to be
saying, there's a single shared subscription queue called queue1,
the first consumer binds *.news the second *.weather
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.topic/*.news; {node: {capabilities: [shared]}, link: {name:
queue1}}"
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.topic/*.weather; {node: {capabilities: [shared]}, link: {name:
queue1}}"
qpid-config -r queues gives
Queue 'queue1'
bind [queue1] => ''
bind [*.news] => amq.topic
bind [*.weather] => amq.topic
For AMQP 0.10 the following would create a similar effect (not sure
if you want auto delete or not, if not remove the x-declare below
and for the AMQP 1.0 example above add reliability: at-least-once
to the link Map)
./drain -b localhost -f \
"queue1; {create: receiver, node: {x-declare:{auto-delete:True},
x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key:
'*.news'}]}}"
./drain -b localhost -f \
"queue1; {create: receiver, node: {x-declare:{auto-delete:True},
x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key:
'*.weather'}]}}"
The following also works for AMQP 0.10
./drain -b localhost -f \
"queue1; {create: receiver, node: {x-declare:{auto-delete:True}},
link: {x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key:
'*.news'}]}}"
./drain -b localhost -f \
"queue1; {create: receiver, node: {x-declare:{auto-delete:True}},
link: {x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key:
'*.weather'}]}}"
Don't know if this is what you are looking for.
Note that in none of the cases above have I worked out how to
remove a binding other than by removing the queue so if you add the
first then the second then delete the second both bindings remain
in place - I did wonder about putting the x-declare/auto delete
stuff on the link in the second AMQP 0.10 example, but that doesn't
seem to remove the binding, so I'm not sure if that's possible.
I need to route the message to the correct application level
destination(s). To do this I need to undertake a matching
operation between the routing key of the message and the binding
key(s) of the created receivers; qpid does not deliver the message
to the receiver with the most exact binding key match.
I guess than I'm not understanding you here. As far as I'm aware if
you've got multiple bindings between an exchange and a queue then
the message will be delivered on to the queue if either binding
matches, so it behaves like a logical OR. In your scenario if the
first receiver adds *.news then the second adds *.weather then from
that point on they will *both* start to receive (*.news OR *.weather)
So basically the receivers, and their bindings, enable the
required messages to get delivered to the required client, but I
then need to undertake application level routing to route the
message to one or more application level classes, based on message
routing key/ receiver binding key matches.
So I'm still totally baffled why you want to send them to the same
queue if you are then demultiplexing at the application level.
Surely (for example) you'd be better having a news queue for the
*.news messages and a weather queue for the *.weather messages. If
you force them down the same queue then you are going to have to do
application level demultiplexing, which it sounds like you don't
want to do, but why use a single queue.
What's actually driving the single queue requirement? That sounds
like the root of your problems, without knowing the nuance of your
scenario it feels like your approaching the problem from the wrong
angle and fighting the middleware rather than letting it work for
you. I'm sure I've missed something subtle in your use case.
Unfortunately in both cases the messaging API does not provide
visibility of the bind matching algorithms and so I have to create
several utility classes to support this functionality.
Would it be possible to create a Binding.h class in the messaging
API to support matching of bindings from all the supported
exchange types?
I'm not actually sure what you are asking for here. Are you asking
for a client side filtering API?
As I say I'm having trouble getting under the skin of your use
case. If I'm reading it correctly it sounds like you are wanting to
have a single queue but have multiple bindings between an exchange
and that queue, which will result in messages for both bindings
making their way on to the queue and then, to get around that, to
apply a client side filter to deliver the right message to the
right receiver - is that correct?
I'm afraid that I'm still not clear why you want to do that on the
client rather than on the broker??
Other's might have a better view, but I'm not sure that client side
filtering fits into the qpid::messaging API per se (and binding
probably wouldn't be a good idea anyway as it's a legacy AMQP 0.10
concept).
One thought moving forward (and I'm far from an expert) might be to
think in terms of AMQP 1.0, so the Qpid Broker may be viewed as
essentially an AMQP 1.0 container and it has a whole bunch of
capabilities, including the ability to filter (the traditional
bindings plus - really cool - message selectors). The
qpid::messaging API is about interacting with nodes on a container
and attaching links with specified properties.
As it happens though an AMQP 1.0 client application can also be
thought of as a container, so an interesting thought might be a
client application containing its own addressable node. In this
scenario you'd establish all the stuff previously discussed with
the broker and the consumer client would have all messages
delivered to the node on the client, you could then (in theory
'cause none of this exists) create AMQP links (on the client) to
the node (on the client) passing filter properties on attachment
(such as a selector).
As I say none of this exists at the moment (except on the broker)
but it might be interesting to consider if it would be possible to
modularise things such that some of these fairly general purpose
AMQP 1.0 "services" could be extracted from the broker and made
available as a toolkit for creating general purpose AMQP 1.0
containers.
As I say I'm no expert and tentatively finding my feet with AMQP
1.0, Gordon Sim would be far better placed than I to say whether
that a) makes sense from an AMQP 1.0 perspective b) how feasible it
is and c) how likely it is to happen :-)
Hope I've managed to be at least some help Clive,
Cheers,
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
