Re: Message API - Real world usage issue
Hi again Clive,
I'm not sure if you've tried the stuff below yet, but I've got some good
news
Using the consumers
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='bill'\"}}"
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='ben'\"}}"
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='tim'\"}}"
I was a bit worried what would happen if messages that don't match get
pushed onto the queue, but I've just tried
./spout -b localhost --content "Hello World" "queue1/fred"
./spout -b localhost --content "Hello World" "queue1/fred"
./spout -b localhost --content "Hello World" "queue1/fred"
./spout -b localhost --content "Hello World" "queue1/fred"
./spout -b localhost --content "Hello World" "queue1/fred"
./spout -b localhost --content "Hello World" "queue1/fred"
./spout -b localhost --content "Hello World" "queue1/fred"
./spout -b localhost --content "Hello World" "queue1/fred"
./spout -b localhost --content "Hello World" "queue1/fred"
./spout -b localhost --content "Hello World" "queue1/fred"
./spout -b localhost --content "Hello World" "queue1/tim"
And that seems to send the last message to the tim receiver quite
happily, which is I guess what you want.
One thing to be aware of though is that I haven't got a clue how message
selectors are implemented when linking to queue nodes. In the case of
exchange nodes they are implemented between the exchange and the
subscription queue so a message doesn't get added to the subscription
queue unless it matches, so that would seem to be a one-off match, but
with selectors on queues I *wonder* if it's behaving more like a browser.
What I mean is that all of the fred messages above remain on the queue
though tim seems to get sent through so it's clearly not simply checking
that the head of the queue matches the selector, which makes me wonder
whether performance degrades as the queue depth increases??
I've got no evidence for that, just a nagging feeling, perhaps someone
more familiar with the implementation would be able to comment.
Regards,
Frase
On 17/02/14 18:13, CLIVE wrote:
Fraze,
Thanks for taking a further look at this. I have not come across the
selector option before. In fact I tried it today at work on a 0.20
version, but got an exception due to this option not being a valid
name. Currently building 0.26 and will give it another go tomorrow.
Just one thought, as the selector is part of the link section, will
this still work if the receivers are all in the same process, using a
single session. I had hoped to try this today but ran out of time
waiting for the 0.26 version to build.
Will be looking at AMQP 1.0 over the next few weeks.
Anyway thanks for all the advice and help so far.
Clive
On 14/02/2014 09:20, Fraser Adams wrote:
Hi again Clive,
A follow up to the mail that I sent yesterday evening. I've got this
thing working using subject in AMQP 0.10 using the following
addresses in drain and spout:
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='bill'\"}}"
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='ben'\"}}"
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='tim'\"}}"
./spout -b localhost --content "Hello World" "queue1/tim"
However one thing I'm still stumped on is what happens with AMQP 1.0.
I tried to do a send to the receivers above using:
./spout --connection-options {protocol:amqp1.0} -b localhost
--content "Hello World" "queue1/tim"
In other words I've only changed the sender protocol to AMQP 1.0 and
now the message doesn't get through!!
It doesn't work either for an AMQP 1.0 consumer with the AMQP 1.0
producer
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='tim'\"}}"
Though the above *does* work with the original AMQP 0.10 send???!!!
I've tried
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"subject='tim'\"}}"
and
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"amqp.subject='tim'\"}}"
but nothing seems to work with the AMQP 1.0 producer example.
Annoyingly, if I simply consume off the queue with no selector by doing:
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver}"
./spout --connection-options {protocol:amqp1.0} -b localhost
--content "Hello World" "queue1/tim"
The message reported by drain says:
Message(properties={spout-id:88fc4a67-d71e-4b86-b967-f897306400e7:0},
subject='tim', content='Hello World')
So *something* thinks th
Re: Message API - Real world usage issue
Frase,
Sorry about the previous z, slip of the old fingers...
As you say, my API is wrong and I will need to think again. Just started
playing with AMQP 1.0, once I have a better understanding of this and
how a Client API might encapsulate both implementations, I will try again.
Thanks for all the help, your AMQP 1.0 user notes were particularly
useful in getting started with the new version.
Regards
Clive
On 17/02/2014 19:12, Fraser Adams wrote:
On 17/02/14 18:13, CLIVE wrote:
Fraze,
It's with an "s" :-)
Thanks for taking a further look at this. I have not come across the
selector option before. In fact I tried it today at work on a 0.20
version, but got an exception due to this option not being a valid
name. Currently building 0.26 and will give it another go tomorrow.
Just one thought, as the selector is part of the link section, will
this still work if the receivers are all in the same process, using a
single session. I had hoped to try this today but ran out of time
waiting for the 0.26 version to build.
To be honest I've not tried this I've only just started messing with
this stuff myself over the last few weeks (you might have seen my post
"A write up of some AMQP 1.0 Experiments") and I've mainly been using
spout and drain for simplicity so I didn't have to write any code :-)
As it happens using Selectors with a queue node wasn't something I'd
tried at all until I saw your post, so I figured I might as well have
a play and was quite pleased that at face value it seems to be doing
something fairly close to your use case :-)
To answer your question though I'm no AMQP 1.0 expert but in the AMQP
1.0 specification section 2.1.2 figure 2.9 illustrates a class diagram
of communication endpoints and that shows multiplicities as follows:
+-+
| Link| Message Transport
+-+ (Node to Node)
| name|
| source |
| target |
| timeout |
+-+
/|\ 0..n
|
|
|
\|/ 0..1
++
| Session| Frame Transport
++ (Container to Container)
| name |
++
/|\ 0..n
|
|
|
\|/ 1..1
++
| Connection | Frame Transport
++ (Container to Container)
| principal |
++
In other words the Session to Link multiplicity is 0..n which
*suggests* that it should "still work if the receivers are all in the
same process, using a single session". Worst case scenario would be
you might need different sessions, but I suspect that it'll probably
be fine. I think that it's just a case of calling createReceiver with
the appropriate address - I think that the receiver in AMQP 1.0 terms
is a terminus, which *does* have an association with the Link unlike
your previous thought process where you assumed that the bindings had
an association with the receiver (which they don't).
Oh one thing that I've not actually tried is what happens if something
is on your single queue *ahead* of things you care about - so what
happens if you have bob/bill/tim selectors and you send a message with
the subject "freddy" ahead of the message with the subject "tim" (if
you see what I mean).
I can't try it out at the moment because I currently can't compile off
trunk 'cause something is broken :-(
You've hopefully also seen my posts that relate to the "subject" - it
works fine in AMQP 0.10 using "qpid.subject" but in AMQP 1.0 it's part
of the immutable properties and has a specific accessor - it should be
fairly easy to add support if people think it's useful though.
Will be looking at AMQP 1.0 over the next few weeks.
Anyway thanks for all the advice and help so far.
Clive
No worries, hope it has been useful - helps me learn more stuff too :-)
FWIW though I'm *still* not convinced that producers delivering
everything to the same queue is the best idea if you've got a mixture
of fast and slow producers writing to the same queue you run a risk of
the fast producers "bullying" the messages of the slower one, I'm not
sure what you have against queues. FWIW even if you configure a queue
with say 2GB capacity that capacity is only used if the queue is
actually full so 100*1MB queues or 1*100MB queue you still need
sufficient capacity to cope with producer burstiness or consumers dying.
In your earlier mail you said:
So at run time you have no way of knowing what address strings are
going to be passed to you.
The User application then makes the following calls on your API
Messaging.registerUserCallback( "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}" ,usrCallback1);
Messaging.registerUserCallback( "queue2; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}" ,usrCallback2);
The User application might not make a call with an address string that
referenc
Re: Message API - Real world usage issue
On 17/02/14 18:13, CLIVE wrote:
Fraze,
It's with an "s" :-)
Thanks for taking a further look at this. I have not come across the
selector option before. In fact I tried it today at work on a 0.20
version, but got an exception due to this option not being a valid
name. Currently building 0.26 and will give it another go tomorrow.
Just one thought, as the selector is part of the link section, will
this still work if the receivers are all in the same process, using a
single session. I had hoped to try this today but ran out of time
waiting for the 0.26 version to build.
To be honest I've not tried this I've only just started messing with
this stuff myself over the last few weeks (you might have seen my post
"A write up of some AMQP 1.0 Experiments") and I've mainly been using
spout and drain for simplicity so I didn't have to write any code :-)
As it happens using Selectors with a queue node wasn't something I'd
tried at all until I saw your post, so I figured I might as well have a
play and was quite pleased that at face value it seems to be doing
something fairly close to your use case :-)
To answer your question though I'm no AMQP 1.0 expert but in the AMQP
1.0 specification section 2.1.2 figure 2.9 illustrates a class diagram
of communication endpoints and that shows multiplicities as follows:
+-+
| Link| Message Transport
+-+ (Node to Node)
| name|
| source |
| target |
| timeout |
+-+
/|\ 0..n
|
|
|
\|/ 0..1
++
| Session| Frame Transport
++ (Container to Container)
| name |
++
/|\ 0..n
|
|
|
\|/ 1..1
++
| Connection | Frame Transport
++ (Container to Container)
| principal |
++
In other words the Session to Link multiplicity is 0..n which *suggests*
that it should "still work if the receivers are all in the same process,
using a single session". Worst case scenario would be you might need
different sessions, but I suspect that it'll probably be fine. I think
that it's just a case of calling createReceiver with the appropriate
address - I think that the receiver in AMQP 1.0 terms is a terminus,
which *does* have an association with the Link unlike your previous
thought process where you assumed that the bindings had an association
with the receiver (which they don't).
Oh one thing that I've not actually tried is what happens if something
is on your single queue *ahead* of things you care about - so what
happens if you have bob/bill/tim selectors and you send a message with
the subject "freddy" ahead of the message with the subject "tim" (if you
see what I mean).
I can't try it out at the moment because I currently can't compile off
trunk 'cause something is broken :-(
You've hopefully also seen my posts that relate to the "subject" - it
works fine in AMQP 0.10 using "qpid.subject" but in AMQP 1.0 it's part
of the immutable properties and has a specific accessor - it should be
fairly easy to add support if people think it's useful though.
Will be looking at AMQP 1.0 over the next few weeks.
Anyway thanks for all the advice and help so far.
Clive
No worries, hope it has been useful - helps me learn more stuff too :-)
FWIW though I'm *still* not convinced that producers delivering
everything to the same queue is the best idea if you've got a mixture of
fast and slow producers writing to the same queue you run a risk of the
fast producers "bullying" the messages of the slower one, I'm not sure
what you have against queues. FWIW even if you configure a queue with
say 2GB capacity that capacity is only used if the queue is actually
full so 100*1MB queues or 1*100MB queue you still need sufficient
capacity to cope with producer burstiness or consumers dying.
In your earlier mail you said:
So at run time you have no way of knowing what address strings are going
to be passed to you.
The User application then makes the following calls on your API
Messaging.registerUserCallback( "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}" ,usrCallback1);
Messaging.registerUserCallback( "queue2; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}" ,usrCallback2);
The User application might not make a call with an address string that
references the same queue, but as a good API designer, you need to be
able to handle it.
I'd argue that in this case you've not *really* got your own API, you've
just added a convenient abstraction around qpid::messaging, in which
case you've not actually really abstracted users away from having to
supply Address Strings that are semantically correct.
As explained in previous responses people supplying the address strings
would *still* need to be aware that
Re: Message API - Real world usage issue
Fraze,
Thanks for taking a further look at this. I have not come across the
selector option before. In fact I tried it today at work on a 0.20
version, but got an exception due to this option not being a valid name.
Currently building 0.26 and will give it another go tomorrow.
Just one thought, as the selector is part of the link section, will this
still work if the receivers are all in the same process, using a single
session. I had hoped to try this today but ran out of time waiting for
the 0.26 version to build.
Will be looking at AMQP 1.0 over the next few weeks.
Anyway thanks for all the advice and help so far.
Clive
On 14/02/2014 09:20, Fraser Adams wrote:
Hi again Clive,
A follow up to the mail that I sent yesterday evening. I've got this
thing working using subject in AMQP 0.10 using the following addresses
in drain and spout:
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='bill'\"}}"
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='ben'\"}}"
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='tim'\"}}"
./spout -b localhost --content "Hello World" "queue1/tim"
However one thing I'm still stumped on is what happens with AMQP 1.0.
I tried to do a send to the receivers above using:
./spout --connection-options {protocol:amqp1.0} -b localhost --content
"Hello World" "queue1/tim"
In other words I've only changed the sender protocol to AMQP 1.0 and
now the message doesn't get through!!
It doesn't work either for an AMQP 1.0 consumer with the AMQP 1.0
producer
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='tim'\"}}"
Though the above *does* work with the original AMQP 0.10 send???!!!
I've tried
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"subject='tim'\"}}"
and
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"amqp.subject='tim'\"}}"
but nothing seems to work with the AMQP 1.0 producer example.
Annoyingly, if I simply consume off the queue with no selector by doing:
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver}"
./spout --connection-options {protocol:amqp1.0} -b localhost --content
"Hello World" "queue1/tim"
The message reported by drain says:
Message(properties={spout-id:88fc4a67-d71e-4b86-b967-f897306400e7:0},
subject='tim', content='Hello World')
So *something* thinks that there's a subject property set, but the
selector doesn't seem to see it.
Does anybody know how to specify the message subject as a property for
a Message Selector in AMQP 1.0???
Regards,
Frase
On 13/02/14 18:04, Fraser Adams wrote:
Hey again Clive,
Hope that you are well.
I think that I might have made a little progress against your use case.
All of the previous comments on this email thread still hold in terms
of what has been said about queues, exchanges, bindings etc. Robbie's
most recent comment "I believe its really only conveyed in the
address string as a form of extension point offering some ability to
leverage the AMQP 0-10 bind commands" is exactly it.
However as Robbie and myself suggested earlier the move towards AMQP
1.0 gives additional exciting options for message filtering in the
form of JMS style Message Selectors.
In your original scenario you tried to do:
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And you were hoping that a message with the subject "tim" would only
end up getting delivered to Rxer 3 and hopefully we've explained why
that isn't the case.
But let's try using Message Selectors. I'm using drain and spout
because I don't need to write any code but it should translate into
what you are doing. I'm using qpid 0.27 compiled off trunk about
three weeks ago.
So starting with qpidd --auth no for simplicity I fire up the
following in three separate windows:
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"test='bill'\"}}"
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"test='ben'\"}}"
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link:
Re: Visibility of amqp subject to Message Selectors - was Re: Message API - Real world usage issue
On 16 February 2014 09:25, Fraser Adams wrote:
> On 15/02/14 21:11, Robbie Gemmell wrote:
>
>> On 14 February 2014 09:20, Fraser Adams > >wrote:
>>
>> The message reported by drain says:
>>>
>>> Message(properties={spout-id:88fc4a67-d71e-4b86-b967-f897306400e7:0},
>>> subject='tim', content='Hello World')
>>>
>>> So *something* thinks that there's a subject property set, but the
>>> selector doesn't seem to see it.
>>>
>>>
>>> I expect that means the 'subject' field of the 'properties' section of
>> the
>> AMQP message is set, and not a 'property' called 'subject' in the
>> 'application-properties' section (which is where the spout-id property is
>> presumably being set).
>>
> Thanks Robbie, yeah this had started to dawn on me yesterday. I was
> looking through the C++ broker code and noticed in
> /cpp/src/qpid/broker/amqp/Message.cpp
>
> std::string Message::getRoutingKey() const
> {
> std::string v;
> v.assign(subject.data, subject.size);
> return v;
> }
>
> which seems to be the only place where subject gets exposed - so it's in a
> specific accessor not a general property one, then I remembered that AMQP
> 1.0 has immutable properties in section 3.2.4 and application properties in
> 3.2.5 and remembered that subject is one of the 3.2.4 properties.
>
> in /qpid-trunk/qpid/specs/apache-filters.xml it explains about
> mapping the JMS headers and most of the JMS headers relate to 3.2.4
> properties.
>
> What I think is happening is that in AMQP 1.0 qpid::messaging used by
> spout actually populates the AMQP 1.0 subject field correctly with the
> subject, but when specifying subject in the Selector that's looking for an
> *application property* called "subject". That'll be why the AMQP 0.10 thing
> works because the legacy "qpid.subject" actually *is* an application
> property.
>
> In /cpp/src/qpid/broker/Selector.cpp there's a method:
> const Value MessageSelectorEnv::specialValue(const string& id) const
>
> That does the mappings mentioned in the apache-filters.xml
>
> the method
>
> const Value& MessageSelectorEnv::value(const string& identifier) const
>
> Checks for the prefix "amqp." and if set calls specialValue() otherwise it
> looks up the application properties.
>
> Of the AMQP 1.0 section 3.2.4 properties it looks like
> user-id
> to
> subject
> reply-to
> content-type
> content-encoding
> group-id
> group-sequence
> reply-to-group-id
>
> Don't have any accessor in the Selector (though to and reply-to have a
> comment "Hard to get this correct for both 1.0 and 0-10" though it might be
> worth exposing them as a LIKE operator might help abstract the differences)
> the group-id, group-sequence and reply-to-group-id don't even appear to be
> extracted in Message.cpp from what I can see.
>
> Anyway I'd *imagine* that it wouldn't be terribly hard to expand the
> pattern used for the JMS Headers to allow access to some of these e.g.
> Selector keys of
> amqp.user_id
> amqp.subject
> amqp.content_type
> amqp.content_encoding
>
> Would make them visible to specialValue() where they could be added to the
> if/else block and their specific accessors could then be called.
>
> I don't know if there is any specific reason for not doing this other than
> the original goal being to map JMS Message Selectors.
>
> From my responses to Clive it would seem to suggest that subject at least
> would be really useful to be exposed to a Selector especially as it's a
> core routing attribute for other filter types.
>
> I'd be interested to know what others think - it looks a fairly simple
> addition.
>
>
>
>>
>> Does anybody know how to specify the message subject as a property for a
>>> Message Selector in AMQP 1.0???
>>>
>>> As the filter in question is specifically only covering functionality
>> required for JMS (specifically mapping the JMS Headers to the relevant
>> fields in the AMQP headers and properties sections, and JMS properties to
>> the AMQP application-properties section) its not clear to me that it
>> allows
>> for selection of the 'subject' field in the 'properties' section, because
>> that isn't used to map any of the JMS headers or property values and so
>> effectively isn't discussed.
>>
>> [snip]
>>
> Yeah as mentioned above that looks about right. I've not looked in the
> Java Broker Selector code, but I imagine that it does something similar to
> the C++ code and looks for the "amqp." prefix but only covers the 3.2.4
> properties mentioned in apache-filters.xml.
>
I wouldnt necessarily assume it supports this filter yet...
>
> To my mind it doesn't seem unreasonable to extend this to subject (at
> least) given that it's a key filtering property for other filters, but it's
> probably worth a discussion.
>
This basically goes back to the earlier discussion in that there are things
the filter can't do, because it didn't (and still doesn't) need to for the
reason it was explicitly created. It may be worth either generalising it to
support more stuff like this or adding another filte
Visibility of amqp subject to Message Selectors - was Re: Message API - Real world usage issue
On 15/02/14 21:11, Robbie Gemmell wrote:
On 14 February 2014 09:20, Fraser Adams wrote:
The message reported by drain says:
Message(properties={spout-id:88fc4a67-d71e-4b86-b967-f897306400e7:0},
subject='tim', content='Hello World')
So *something* thinks that there's a subject property set, but the
selector doesn't seem to see it.
I expect that means the 'subject' field of the 'properties' section of the
AMQP message is set, and not a 'property' called 'subject' in the
'application-properties' section (which is where the spout-id property is
presumably being set).
Thanks Robbie, yeah this had started to dawn on me yesterday. I was
looking through the C++ broker code and noticed in
/cpp/src/qpid/broker/amqp/Message.cpp
std::string Message::getRoutingKey() const
{
std::string v;
v.assign(subject.data, subject.size);
return v;
}
which seems to be the only place where subject gets exposed - so it's in
a specific accessor not a general property one, then I remembered that
AMQP 1.0 has immutable properties in section 3.2.4 and application
properties in 3.2.5 and remembered that subject is one of the 3.2.4
properties.
in /qpid-trunk/qpid/specs/apache-filters.xml it explains about
mapping the JMS headers and most of the JMS headers relate to 3.2.4
properties.
What I think is happening is that in AMQP 1.0 qpid::messaging used by
spout actually populates the AMQP 1.0 subject field correctly with the
subject, but when specifying subject in the Selector that's looking for
an *application property* called "subject". That'll be why the AMQP 0.10
thing works because the legacy "qpid.subject" actually *is* an
application property.
In /cpp/src/qpid/broker/Selector.cpp there's a method:
const Value MessageSelectorEnv::specialValue(const string& id) const
That does the mappings mentioned in the apache-filters.xml
the method
const Value& MessageSelectorEnv::value(const string& identifier) const
Checks for the prefix "amqp." and if set calls specialValue() otherwise
it looks up the application properties.
Of the AMQP 1.0 section 3.2.4 properties it looks like
user-id
to
subject
reply-to
content-type
content-encoding
group-id
group-sequence
reply-to-group-id
Don't have any accessor in the Selector (though to and reply-to have a
comment "Hard to get this correct for both 1.0 and 0-10" though it might
be worth exposing them as a LIKE operator might help abstract the
differences) the group-id, group-sequence and reply-to-group-id don't
even appear to be extracted in Message.cpp from what I can see.
Anyway I'd *imagine* that it wouldn't be terribly hard to expand the
pattern used for the JMS Headers to allow access to some of these e.g.
Selector keys of
amqp.user_id
amqp.subject
amqp.content_type
amqp.content_encoding
Would make them visible to specialValue() where they could be added to
the if/else block and their specific accessors could then be called.
I don't know if there is any specific reason for not doing this other
than the original goal being to map JMS Message Selectors.
From my responses to Clive it would seem to suggest that subject at
least would be really useful to be exposed to a Selector especially as
it's a core routing attribute for other filter types.
I'd be interested to know what others think - it looks a fairly simple
addition.
Does anybody know how to specify the message subject as a property for a
Message Selector in AMQP 1.0???
As the filter in question is specifically only covering functionality
required for JMS (specifically mapping the JMS Headers to the relevant
fields in the AMQP headers and properties sections, and JMS properties to
the AMQP application-properties section) its not clear to me that it allows
for selection of the 'subject' field in the 'properties' section, because
that isn't used to map any of the JMS headers or property values and so
effectively isn't discussed.
[snip]
Yeah as mentioned above that looks about right. I've not looked in the
Java Broker Selector code, but I imagine that it does something similar
to the C++ code and looks for the "amqp." prefix but only covers the
3.2.4 properties mentioned in apache-filters.xml.
To my mind it doesn't seem unreasonable to extend this to subject (at
least) given that it's a key filtering property for other filters, but
it's probably worth a discussion.
Cheers,
Frase
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
Re: Message API - Real world usage issue
On 14 February 2014 09:20, Fraser Adams wrote:
> Hi again Clive,
> A follow up to the mail that I sent yesterday evening. I've got this thing
> working using subject in AMQP 0.10 using the following addresses in drain
> and spout:
>
>
> ./drain -b localhost -f \
> "queue1; {create: receiver, link: {name: test-link, selector:
> \"qpid.subject='bill'\"}}"
>
> ./drain -b localhost -f \
> "queue1; {create: receiver, link: {name: test-link, selector:
> \"qpid.subject='ben'\"}}"
>
> ./drain -b localhost -f \
> "queue1; {create: receiver, link: {name: test-link, selector:
> \"qpid.subject='tim'\"}}"
>
>
> ./spout -b localhost --content "Hello World" "queue1/tim"
>
>
>
> However one thing I'm still stumped on is what happens with AMQP 1.0. I
> tried to do a send to the receivers above using:
>
> ./spout --connection-options {protocol:amqp1.0} -b localhost --content
> "Hello World" "queue1/tim"
>
> In other words I've only changed the sender protocol to AMQP 1.0 and now
> the message doesn't get through!!
>
>
> It doesn't work either for an AMQP 1.0 consumer with the AMQP 1.0 producer
>
>
> ./drain --connection-options {protocol:amqp1.0} -b localhost -f \
> "queue1; {create: receiver, link: {name: test-link, selector:
> \"qpid.subject='tim'\"}}"
>
> Though the above *does* work with the original AMQP 0.10 send???!!!
>
> I've tried
>
>
> ./drain --connection-options {protocol:amqp1.0} -b localhost -f \
> "queue1; {create: receiver, link: {name: test-link, selector:
> \"subject='tim'\"}}"
>
> and
>
>
> ./drain --connection-options {protocol:amqp1.0} -b localhost -f \
> "queue1; {create: receiver, link: {name: test-link, selector:
> \"amqp.subject='tim'\"}}"
>
> but nothing seems to work with the AMQP 1.0 producer example.
>
>
> Annoyingly, if I simply consume off the queue with no selector by doing:
>
>
> ./drain --connection-options {protocol:amqp1.0} -b localhost -f \
> "queue1; {create: receiver}"
>
> ./spout --connection-options {protocol:amqp1.0} -b localhost --content
> "Hello World" "queue1/tim"
>
> The message reported by drain says:
>
> Message(properties={spout-id:88fc4a67-d71e-4b86-b967-f897306400e7:0},
> subject='tim', content='Hello World')
>
> So *something* thinks that there's a subject property set, but the
> selector doesn't seem to see it.
>
>
I expect that means the 'subject' field of the 'properties' section of the
AMQP message is set, and not a 'property' called 'subject' in the
'application-properties' section (which is where the spout-id property is
presumably being set).
>
> Does anybody know how to specify the message subject as a property for a
> Message Selector in AMQP 1.0???
>
As the filter in question is specifically only covering functionality
required for JMS (specifically mapping the JMS Headers to the relevant
fields in the AMQP headers and properties sections, and JMS properties to
the AMQP application-properties section) its not clear to me that it allows
for selection of the 'subject' field in the 'properties' section, because
that isn't used to map any of the JMS headers or property values and so
effectively isn't discussed.
>
> Regards,
> Frase
>
>
> On 13/02/14 18:04, Fraser Adams wrote:
>
>> Hey again Clive,
>> Hope that you are well.
>>
>> I think that I might have made a little progress against your use case.
>>
>> All of the previous comments on this email thread still hold in terms of
>> what has been said about queues, exchanges, bindings etc. Robbie's most
>> recent comment "I believe its really only conveyed in the address string as
>> a form of extension point offering some ability to leverage the AMQP 0-10
>> bind commands" is exactly it.
>>
>> However as Robbie and myself suggested earlier the move towards AMQP 1.0
>> gives additional exciting options for message filtering in the form of JMS
>> style Message Selectors.
>>
>> In your original scenario you tried to do:
>>
>>Rxer 1 - "queue1; {create: receiver, node:
>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>> 'queue1', key: 'bill'}]}}"
>>Rxer 2 - "queue1; {create: receiver, node:
>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>> 'queue1', key: 'ben'}]}}"
>>Rxer 3 - "queue1; {create: receiver, node:
>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>> 'queue1', key: 'tim'}]}}"
>>
>> And you were hoping that a message with the subject "tim" would only end
>> up getting delivered to Rxer 3 and hopefully we've explained why that isn't
>> the case.
>>
>> But let's try using Message Selectors. I'm using drain and spout because
>> I don't need to write any code but it should translate into what you are
>> doing. I'm using qpid 0.27 compiled off trunk about three weeks ago.
>>
>> So starting with qpidd --auth no for simplicity I fire up the following
>> in three separate windows:
>>
>> ./drain --connection-options {protocol:amqp1.0} -b localhost -f \
>> "queue1; {create: receiver, link: {name: test-link, selector:
Re: Message API - Real world usage issue
Hi again Clive,
A follow up to the mail that I sent yesterday evening. I've got this
thing working using subject in AMQP 0.10 using the following addresses
in drain and spout:
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='bill'\"}}"
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='ben'\"}}"
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='tim'\"}}"
./spout -b localhost --content "Hello World" "queue1/tim"
However one thing I'm still stumped on is what happens with AMQP 1.0. I
tried to do a send to the receivers above using:
./spout --connection-options {protocol:amqp1.0} -b localhost --content
"Hello World" "queue1/tim"
In other words I've only changed the sender protocol to AMQP 1.0 and now
the message doesn't get through!!
It doesn't work either for an AMQP 1.0 consumer with the AMQP 1.0 producer
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"qpid.subject='tim'\"}}"
Though the above *does* work with the original AMQP 0.10 send???!!!
I've tried
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"subject='tim'\"}}"
and
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"amqp.subject='tim'\"}}"
but nothing seems to work with the AMQP 1.0 producer example.
Annoyingly, if I simply consume off the queue with no selector by doing:
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver}"
./spout --connection-options {protocol:amqp1.0} -b localhost --content
"Hello World" "queue1/tim"
The message reported by drain says:
Message(properties={spout-id:88fc4a67-d71e-4b86-b967-f897306400e7:0},
subject='tim', content='Hello World')
So *something* thinks that there's a subject property set, but the
selector doesn't seem to see it.
Does anybody know how to specify the message subject as a property for a
Message Selector in AMQP 1.0???
Regards,
Frase
On 13/02/14 18:04, Fraser Adams wrote:
Hey again Clive,
Hope that you are well.
I think that I might have made a little progress against your use case.
All of the previous comments on this email thread still hold in terms
of what has been said about queues, exchanges, bindings etc. Robbie's
most recent comment "I believe its really only conveyed in the address
string as a form of extension point offering some ability to leverage
the AMQP 0-10 bind commands" is exactly it.
However as Robbie and myself suggested earlier the move towards AMQP
1.0 gives additional exciting options for message filtering in the
form of JMS style Message Selectors.
In your original scenario you tried to do:
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And you were hoping that a message with the subject "tim" would only
end up getting delivered to Rxer 3 and hopefully we've explained why
that isn't the case.
But let's try using Message Selectors. I'm using drain and spout
because I don't need to write any code but it should translate into
what you are doing. I'm using qpid 0.27 compiled off trunk about three
weeks ago.
So starting with qpidd --auth no for simplicity I fire up the
following in three separate windows:
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"test='bill'\"}}"
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"test='ben'\"}}"
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"test='tim'\"}}"
What this is doing is to start up three separate AMQP 1.0 consumers
that all consume from queue1 and have Message Selectors that inspect
the property "test" for the values "bill", "ben" and "tim" respectively.
If you do qpid-config -r queues you'll see
Queue 'queue1'
bind [queue1] => ''
I then do:
./spout --connection-options {protocol:amqp1.0} -b localhost --content
"Hello World" -P test=tim "queue1"
Which sends a message with the text "Hello World" and the property
"test" set to "tim" to the node addressed "queue1" e.g. to the queue
queue1 in this case.
When I do this happily what I see is nothing in the bill a
Re: Message API - Real world usage issue
Hey again Clive,
Hope that you are well.
I think that I might have made a little progress against your use case.
All of the previous comments on this email thread still hold in terms of
what has been said about queues, exchanges, bindings etc. Robbie's most
recent comment "I believe its really only conveyed in the address string
as a form of extension point offering some ability to leverage the AMQP
0-10 bind commands" is exactly it.
However as Robbie and myself suggested earlier the move towards AMQP 1.0
gives additional exciting options for message filtering in the form of
JMS style Message Selectors.
In your original scenario you tried to do:
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And you were hoping that a message with the subject "tim" would only end
up getting delivered to Rxer 3 and hopefully we've explained why that
isn't the case.
But let's try using Message Selectors. I'm using drain and spout because
I don't need to write any code but it should translate into what you are
doing. I'm using qpid 0.27 compiled off trunk about three weeks ago.
So starting with qpidd --auth no for simplicity I fire up the following
in three separate windows:
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"test='bill'\"}}"
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"test='ben'\"}}"
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"test='tim'\"}}"
What this is doing is to start up three separate AMQP 1.0 consumers that
all consume from queue1 and have Message Selectors that inspect the
property "test" for the values "bill", "ben" and "tim" respectively.
If you do qpid-config -r queues you'll see
Queue 'queue1'
bind [queue1] => ''
I then do:
./spout --connection-options {protocol:amqp1.0} -b localhost --content
"Hello World" -P test=tim "queue1"
Which sends a message with the text "Hello World" and the property
"test" set to "tim" to the node addressed "queue1" e.g. to the queue
queue1 in this case.
When I do this happily what I see is nothing in the bill and ben windows
and
Message(properties={spout-id:aa92e8c4-3d12-40f6-bc89-8cabe91500e6:0,
test:tim}, content='Hello World')
in the tim window.
I sent the message 20 times and each time it only arrived on the "tim"
receiver.
Although Message Selectors were added as part of the AMQP 1.0 work it
seems that it also works in AMQP 0.10 because when I tried the following
it worked too.
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"test='bill'\"}}"
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"test='ben'\"}}"
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"test='tim'\"}}"
./spout -b localhost --content "Hello World" -P test=tim "queue1"
So it looks like it's *nearly* what you are looking for. One thing that
I've *not* got working yet though is using the subject. I thought that
./drain -b localhost -f \
"queue1; {create: receiver, link: {name: test-link, selector:
\"subject='tim'\"}}"
./spout -b localhost --content "Hello World" "queue1/tim"
would work, but it doesn't seem to. It's likely to be something quirky
like the subject property needing some prefix or other, I don't know. If
I get a moment I'll have a look - or perhaps someone else may know the
answer.
Hope that helps. Do bear in mind that I've literally only spend 45 mins
or so messing with this so there may be gotchas and I've not
investigated the relative pros and cons of using a selector as a filter
off a queue node versus the more traditional topic style subscriptions
(you certainly run the risk of filling your queue up if "tim" stops
consuming though that might be fine if you use a circular queue). I
don't know the earliest broker version where this will work I *think*
Message Selectors were first added to 0.20 but you'd be best to use
something more recent if you want to try this approach.
I hope this helps a bit,
Frase
On 11/02/14 21:57, CLIVE wrote:
Robbie,
Thanks for the response.
You are confirming what Fraze has said, so I obviously need to take
this on board and rethink my understanding of the Receiver concept.
In my mind I had a Receiver as an entity that received the messages
specified by the address string, but in fact I
Re: Message API - Real world usage issue
On 11 February 2014 21:57, CLIVE wrote:
> Robbie,
>
> Thanks for the response.
>
> You are confirming what Fraze has said, so I obviously need to take this
> on board and rethink my understanding of the Receiver concept.
>
> In my mind I had a Receiver as an entity that received the messages
> specified by the address string, but in fact I need to just look at it as a
> conduit to a queue that has one or more bindings associated with it.
>
>
It is basically able to be both of those things, but in this case you are
using it strictly as the latter.
The main issue is simply that the x-bindings declarations you are using
(e.g. x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: 'tim'}] )
aren't really related to the consumer but rather with the bindings between
the exchange and the queue they are receiving from. I believe its really
only conveyed in the address string as a form of extension point offering
some ability to leverage the AMQP 0-10 bind commands in a way that doesn't
require the addition of a very protocol-specific API method such as those
the older client API presumably had. I expect thats why it has a name
beginning "x-", though admit I dont know for sure, perhaps those with
deeper knowledge of the messaging API can speak to that.
>
>
> On 11/02/2014 21:35, Robbie Gemmell wrote:
>
>> On 11 February 2014 19:34, CLIVE wrote:
>>
>> Ted,
>>>
>>> Thanks for the response and your comments.
>>>
>>> I have had to handle the case of multiple Receivers attaching to the same
>>> queue on several occasions; primarily because the customer has felt that
>>> it
>>> was easier to handle one queue with multiple bindings (up to 100), rather
>>> than having a hundred queues with single bindings; message order was
>>> also a
>>> contributing factor.
>>>
>>> The point of the post was just to raise it as a possible issue for future
>>> improvement..
>>>
>>> I carried out a straw poll of 10 developers today at work. I gave them
>>> the
>>> two examples previously described and asked what they would expect to
>>> happen for the case where multiple Receivers were created for the same
>>> queue. They all expected the correct Receiver to be returned from the
>>> nextReceiver method, not the undeterministic behaviour that they would
>>> see.
>>>
>>> As has been mentioned by others, the behaviour you are seeing is
>> expected
>> because it is exactly what you are really asking the client and broker to
>> do currently: one queue which can receive messages via multiple binding
>> keys that have been added, and distribute them to any of the completely
>> equal multiple consumers receiving from it.
>>
>> As Fraser has also beaten me to saying, if you really want to make
>> particular consumers only get particular messages from a shared queue,
>> then
>> you will likely need to look at using selectors so that they can in fact
>> only receive those messages.
>>
>>
>> I wouldn't have thought that it would take that much code/effort to add
>>> some additional functionality in the messaging API Implementation to
>>> support the behavior that, it would appear, most developers would expect
>>> to
>>> see. If I find some time I will take a look and see how it could be done.
>>>
>>>
>>> As Fraser mentioned, I think there is some confusion as to what your
>> reciever creation calls are actually doing, but even if removing that
>> confusion from the equation the situation is not necessarily as simple as
>> it may seem. Suppose two receivers add the same binding key, which is the
>> 'correct' receiver to get the single message? Suppose wildcard matching is
>> in use on the bindings and multiple bindings then match a particular
>> message published, which receiver gets the single message? Imagine
>> selectors are also in use, but mutliple consumers selectors match the
>> message, which reciever gets the message? The list goes on...
>>
>> You are effectively talking about turning the client into a sort of
>> broker,
>> and since you already have one of those its probably easier to just ask it
>> to do what you actually want.
>>
>> Robbie
>>
>>
>>
>>> On 10/02/2014 22:17, Ted Ross wrote:
>>>
>>> Clive,
What you are observing is what I expect: In the second scenario where
you use the same queue for each of the three receivers, the receiver
that
receives any particular message will be non-deterministic.
This is because the binding key is applied between the exchange and the
queue (i.e. it is used to determine which queue(s) the message should be
enqueued on). Multiple receivers on a queue will receive messages from
the
queue in an undetermined order, but no message shall be delivered to
more
than one receiver. In the second case, all of the messages are placed
on
the same queue in the order in which they arrive. The queue acts as a
buffer between the routing rule that matched the message and the
receiver
that provided the routing rule.
Re: Message API - Real world usage issue
Robbie,
Thanks for the response.
You are confirming what Fraze has said, so I obviously need to take this
on board and rethink my understanding of the Receiver concept.
In my mind I had a Receiver as an entity that received the messages
specified by the address string, but in fact I need to just look at it
as a conduit to a queue that has one or more bindings associated with it.
Clive
On 11/02/2014 21:35, Robbie Gemmell wrote:
On 11 February 2014 19:34, CLIVE wrote:
Ted,
Thanks for the response and your comments.
I have had to handle the case of multiple Receivers attaching to the same
queue on several occasions; primarily because the customer has felt that it
was easier to handle one queue with multiple bindings (up to 100), rather
than having a hundred queues with single bindings; message order was also a
contributing factor.
The point of the post was just to raise it as a possible issue for future
improvement..
I carried out a straw poll of 10 developers today at work. I gave them the
two examples previously described and asked what they would expect to
happen for the case where multiple Receivers were created for the same
queue. They all expected the correct Receiver to be returned from the
nextReceiver method, not the undeterministic behaviour that they would see.
As has been mentioned by others, the behaviour you are seeing is expected
because it is exactly what you are really asking the client and broker to
do currently: one queue which can receive messages via multiple binding
keys that have been added, and distribute them to any of the completely
equal multiple consumers receiving from it.
As Fraser has also beaten me to saying, if you really want to make
particular consumers only get particular messages from a shared queue, then
you will likely need to look at using selectors so that they can in fact
only receive those messages.
I wouldn't have thought that it would take that much code/effort to add
some additional functionality in the messaging API Implementation to
support the behavior that, it would appear, most developers would expect to
see. If I find some time I will take a look and see how it could be done.
As Fraser mentioned, I think there is some confusion as to what your
reciever creation calls are actually doing, but even if removing that
confusion from the equation the situation is not necessarily as simple as
it may seem. Suppose two receivers add the same binding key, which is the
'correct' receiver to get the single message? Suppose wildcard matching is
in use on the bindings and multiple bindings then match a particular
message published, which receiver gets the single message? Imagine
selectors are also in use, but mutliple consumers selectors match the
message, which reciever gets the message? The list goes on...
You are effectively talking about turning the client into a sort of broker,
and since you already have one of those its probably easier to just ask it
to do what you actually want.
Robbie
On 10/02/2014 22:17, Ted Ross wrote:
Clive,
What you are observing is what I expect: In the second scenario where
you use the same queue for each of the three receivers, the receiver that
receives any particular message will be non-deterministic.
This is because the binding key is applied between the exchange and the
queue (i.e. it is used to determine which queue(s) the message should be
enqueued on). Multiple receivers on a queue will receive messages from the
queue in an undetermined order, but no message shall be delivered to more
than one receiver. In the second case, all of the messages are placed on
the same queue in the order in which they arrive. The queue acts as a
buffer between the routing rule that matched the message and the receiver
that provided the routing rule.
It would be simpler to do the following:
Rxer 1 - "amq.topic/bill; {link: {x-declare: {auto-delete:true}}}"
Rxer 2 - "amq.topic/ben; {link: {x-declare: {auto-delete:true}}}"
Rxer 3 - "amq.topic/tim; {link: {x-declare: {auto-delete:true}}}"
This will give you the determinism you want. This will cause the
creation of a temporary queue for each receiver that will receive the
messages that match the topic key (following the slash in the address).
-Ted
On 02/10/2014 04:39 PM, CLIVE wrote:
Fraser,
Thanks for the response. The real problem is that the behavior of a
Receiver is different depending on the multiplicity of the binding strategy
used. If you use a single queue with a single binding then messages will
get delivered to the required receiver. If you use multiple Receivers bound
to the same queue, the Receiver called by the messaging API, when
delivering a message to your application, may not be the one that you
think!!
So if I create three Receivers in the same application, with the
following bindings (note unique queue names)
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
Re: Message API - Real world usage issue
On 11 February 2014 21:26, CLIVE wrote:
> On 11/02/2014 19:35, Fraser Adams wrote:
>
>> Clive,
>> In your second scenario I don't see why you'd expect the message to get
>> delivered to Rxer 3.
>>
> I suppose because the Receiver I used with that address had the 'tim'
> binding, and this matches the subject of the message that was sent.
>
>
>> What you've done is deliberately create three bindings to the same queue,
>> so you'd have a match bill OR ben OR tim. If you sent a message with a
>> subject of tim it would simply get added do the queue (that all receivers
>> are consuming from) so it would simply arrive at the next available
>> receiver.
>>
>
> Yes, I realize that the bindings are essentially being OR'ed together. But
> when the message gets pulled off the queue and delivered to the client, why
> make an arbitrary choice about which Receiver to send it to, why not, for a
> bit of extra code, send it to the Receiver that provided the matching
> binding (Although I have to admit that this does raise the question of what
> to do if two or more bindings match??)
>
>
>> Basically the three receivers have just been configured to receive
>> exactly the same thing. If you do qpid-config -r queues you'll see queue1
>> with three bindings.
>>
> No, the receivers are not configured to receiver the same thing, they are
> configured to receive off the same queue.
The receivers are configured to recieve the same thing, everything on the
shared queue they are all receiving from. They also happen to be confgured
to create bindings upon their creation, but those concepts are entirely
distinct.
What you are describing would seem to be message selection, such that a
particular consumer can only receive particular messages on the shared
queue. You aren't using that functionality, but you potentially could in
order to achieve the behaviour you desire.
>
>> I think that you are confusing yourself because it might look like the
>> x-bindings relate to the receiver given how the Address string looks, but
>> as I say you're establishing bindings between amq.topic and queue1 and
>> doing what you've done creates three ORed bindings.
>>
> I don't think I am confusing myself, I understand exactly what is going
> on. In my mind I am creating three high level Receivers, not three ORed
> bindings. The straw poll I carried out at work also seemed to suggest that
> what I would like to see is also what a lot of other developers would
> expect as well.
>
>
>> I'm still struggling to see why you want to use a single queue, when it
>> seems like you actually want to demultiplex.
>>
> As I said customer wanted the single queue to help with message ordering
> and reduce perceived maintenance associated with lots of queues
>
>
>> Can you explain a bit more about what you are actually trying to achieve?
>>
>
> Lets say you provide the following interface method on a public C++ API,
> that shields the User Application from the underlying QPID implementation.
>
> class Messaging {
> public:
> static void registerUserCallback(const std::string& address, const
> boost::function< void (const std::string&) >& callback);
> };
>
> So at run time you have no way of knowing what address strings are going
> to be passed to you.
>
> The User application then makes the following calls on your API
>
> Messaging.registerUserCallback( "queue1; {create: receiver, node:
> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
> 'queue1', key: 'bill'}]}}" ,usrCallback1);
> Messaging.registerUserCallback( "queue2; {create: receiver, node:
> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
> 'queue1', key: 'ben'}]}}" ,usrCallback2);
>
> The User application might not make a call with an address string that
> references the same queue, but as a good API designer, you need to be able
> to handle it.
>
> If the QPID messaging implementation routed messages to the correct
> Receiver, then the implementation is pretty trivial. As it doesn't, then
> the implementation becomes a lot harder.
>
The client is delivering the message to the receiver associated with the
underlying consumer that the broker gave the message to. It isnt really
free to hand it to anything else. The main choices would appear to be to
configure the receivers (and/or queues) such that they can only get the
messages you want them to, or perform application-level distribution, as it
sounds like you have previously done.
>
> Anyway I hope this helps to explain the scenario a little bit better.
>
> Clive
>
>
> Frase
>>
>>
>> On 10/02/14 21:39, CLIVE wrote:
>>
>>> Fraser,
>>>
>>> Thanks for the response. The real problem is that the behavior of a
>>> Receiver is different depending on the multiplicity of the binding strategy
>>> used. If you use a single queue with a single binding then messages will
>>> get delivered to the required receiver. If you use multiple Receivers bound
>>> to the same queue, the Receiver called by the
Re: Message API - Real world usage issue
On 11 February 2014 19:34, CLIVE wrote:
> Ted,
>
> Thanks for the response and your comments.
>
> I have had to handle the case of multiple Receivers attaching to the same
> queue on several occasions; primarily because the customer has felt that it
> was easier to handle one queue with multiple bindings (up to 100), rather
> than having a hundred queues with single bindings; message order was also a
> contributing factor.
>
> The point of the post was just to raise it as a possible issue for future
> improvement..
>
> I carried out a straw poll of 10 developers today at work. I gave them the
> two examples previously described and asked what they would expect to
> happen for the case where multiple Receivers were created for the same
> queue. They all expected the correct Receiver to be returned from the
> nextReceiver method, not the undeterministic behaviour that they would see.
>
As has been mentioned by others, the behaviour you are seeing is expected
because it is exactly what you are really asking the client and broker to
do currently: one queue which can receive messages via multiple binding
keys that have been added, and distribute them to any of the completely
equal multiple consumers receiving from it.
As Fraser has also beaten me to saying, if you really want to make
particular consumers only get particular messages from a shared queue, then
you will likely need to look at using selectors so that they can in fact
only receive those messages.
>
> I wouldn't have thought that it would take that much code/effort to add
> some additional functionality in the messaging API Implementation to
> support the behavior that, it would appear, most developers would expect to
> see. If I find some time I will take a look and see how it could be done.
>
>
As Fraser mentioned, I think there is some confusion as to what your
reciever creation calls are actually doing, but even if removing that
confusion from the equation the situation is not necessarily as simple as
it may seem. Suppose two receivers add the same binding key, which is the
'correct' receiver to get the single message? Suppose wildcard matching is
in use on the bindings and multiple bindings then match a particular
message published, which receiver gets the single message? Imagine
selectors are also in use, but mutliple consumers selectors match the
message, which reciever gets the message? The list goes on...
You are effectively talking about turning the client into a sort of broker,
and since you already have one of those its probably easier to just ask it
to do what you actually want.
Robbie
>
>
> On 10/02/2014 22:17, Ted Ross wrote:
>
>> Clive,
>>
>> What you are observing is what I expect: In the second scenario where
>> you use the same queue for each of the three receivers, the receiver that
>> receives any particular message will be non-deterministic.
>>
>> This is because the binding key is applied between the exchange and the
>> queue (i.e. it is used to determine which queue(s) the message should be
>> enqueued on). Multiple receivers on a queue will receive messages from the
>> queue in an undetermined order, but no message shall be delivered to more
>> than one receiver. In the second case, all of the messages are placed on
>> the same queue in the order in which they arrive. The queue acts as a
>> buffer between the routing rule that matched the message and the receiver
>> that provided the routing rule.
>>
>> It would be simpler to do the following:
>>
>> Rxer 1 - "amq.topic/bill; {link: {x-declare: {auto-delete:true}}}"
>> Rxer 2 - "amq.topic/ben; {link: {x-declare: {auto-delete:true}}}"
>> Rxer 3 - "amq.topic/tim; {link: {x-declare: {auto-delete:true}}}"
>>
>> This will give you the determinism you want. This will cause the
>> creation of a temporary queue for each receiver that will receive the
>> messages that match the topic key (following the slash in the address).
>>
>> -Ted
>>
>>
>> On 02/10/2014 04:39 PM, CLIVE wrote:
>>
>>> Fraser,
>>>
>>> Thanks for the response. The real problem is that the behavior of a
>>> Receiver is different depending on the multiplicity of the binding strategy
>>> used. If you use a single queue with a single binding then messages will
>>> get delivered to the required receiver. If you use multiple Receivers bound
>>> to the same queue, the Receiver called by the messaging API, when
>>> delivering a message to your application, may not be the one that you
>>> think!!
>>>
>>> So if I create three Receivers in the same application, with the
>>> following bindings (note unique queue names)
>>>
>>>Rxer 1 - "queue1; {create: receiver, node:
>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>> 'queue1', key: 'bill'}]}}"
>>>Rxer 2 - "queue2; {create: receiver, node:
>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>> 'queue1', key: 'ben'}]}}"
>>>Rxer 3 - "queue3; {create: receiver, node:
>>> {x-declare:{auto-del
Re: Message API - Real world usage issue
On 11/02/2014 19:35, Fraser Adams wrote:
Clive,
In your second scenario I don't see why you'd expect the message to
get delivered to Rxer 3.
I suppose because the Receiver I used with that address had the 'tim'
binding, and this matches the subject of the message that was sent.
What you've done is deliberately create three bindings to the same
queue, so you'd have a match bill OR ben OR tim. If you sent a message
with a subject of tim it would simply get added do the queue (that all
receivers are consuming from) so it would simply arrive at the next
available receiver.
Yes, I realize that the bindings are essentially being OR'ed together.
But when the message gets pulled off the queue and delivered to the
client, why make an arbitrary choice about which Receiver to send it to,
why not, for a bit of extra code, send it to the Receiver that provided
the matching binding (Although I have to admit that this does raise the
question of what to do if two or more bindings match??)
Basically the three receivers have just been configured to receive
exactly the same thing. If you do qpid-config -r queues you'll see
queue1 with three bindings.
No, the receivers are not configured to receiver the same thing, they
are configured to receive off the same queue.
I think that you are confusing yourself because it might look like the
x-bindings relate to the receiver given how the Address string looks,
but as I say you're establishing bindings between amq.topic and queue1
and doing what you've done creates three ORed bindings.
I don't think I am confusing myself, I understand exactly what is going
on. In my mind I am creating three high level Receivers, not three ORed
bindings. The straw poll I carried out at work also seemed to suggest
that what I would like to see is also what a lot of other developers
would expect as well.
I'm still struggling to see why you want to use a single queue, when
it seems like you actually want to demultiplex.
As I said customer wanted the single queue to help with message ordering
and reduce perceived maintenance associated with lots of queues
Can you explain a bit more about what you are actually trying to achieve?
Lets say you provide the following interface method on a public C++ API,
that shields the User Application from the underlying QPID implementation.
class Messaging {
public:
static void registerUserCallback(const std::string& address,
const boost::function< void (const std::string&) >& callback);
};
So at run time you have no way of knowing what address strings are going
to be passed to you.
The User application then makes the following calls on your API
Messaging.registerUserCallback( "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}" ,usrCallback1);
Messaging.registerUserCallback( "queue2; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}" ,usrCallback2);
The User application might not make a call with an address string that
references the same queue, but as a good API designer, you need to be
able to handle it.
If the QPID messaging implementation routed messages to the correct
Receiver, then the implementation is pretty trivial. As it doesn't, then
the implementation becomes a lot harder.
Anyway I hope this helps to explain the scenario a little bit better.
Clive
Frase
On 10/02/14 21:39, CLIVE wrote:
Fraser,
Thanks for the response. The real problem is that the behavior of a
Receiver is different depending on the multiplicity of the binding
strategy used. If you use a single queue with a single binding then
messages will get delivered to the required receiver. If you use
multiple Receivers bound to the same queue, the Receiver called by
the messaging API, when delivering a message to your application, may
not be the one that you think!!
So if I create three Receivers in the same application, with the
following bindings (note unique queue names)
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue2; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue3; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And then send a message on the amq.topic exchange with a subject of
'tim'. Then Rxer3 will get returned by the 'nextReceiver' method on
the associated Session object.
But if I change the bindings so they related to the same queue
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue1; {create: receiver, node:
{x-declare:{auto-d
Re: Message API - Real world usage issue
I still think you are looking at it the wrong way Clive.
so consider your scenario
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
now think about it a different way and imagine adding the queue and the
bindings via qpid-config and receiving via say:
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}}}"
Rxer 2 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}}}"
Rxer 3 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}}}"
In that case I doubt you'd have any issue assuming that the receivers
would likely alternately pick the next message off the queue.
The reality is that with the receivers that you've specified that's
*exactly* what you've done.
In AMQP terms you are addressing the queue node named queue1 from all
three receivers.
I'm not *quite* sure why your customers have an aversion to queues,
though I guess that the take up memory so perhaps that's why?
I've not thought too hard about this, but I wonder if there's a way to
achieve something like you seem to be trying to achieve via a queue
browser - possibly not in AMQP 0.10 but perhaps in AMQP 1.0 with message
selectors. TBH I haven't played too much with browsers.
I *think* what you seem to be expecting is that the filter is associated
with the receiver, but it's not. In AMQP 0.10 bindings are formed
between exchanges and queues, so there's no way to specify an
association between a particular binding and a given receiver without
also having a queue associated with the receiver - unless you apply your
filtering on the client side.
As I said in my earlier response I think that the correct way to achieve
what amounts to client side filtering would be to expose some of the
logic of qpidd such that it could be used as building blocks for a more
general AMQP 1.0 container then you could do a link attach from your
actual receivers to the client side node in your client container, but
TBH none of that has been considered yet, but I'm not convinced that
what you are suggesting actually maps to either AMQP 0.10 or AMQP 1.0
Re "to support the behavior that, it would appear, most developers would
expect to see" so I'm not convinced that it really is what most
developers would expect if they really thought about it. If you replay
the question to them in the way I suggested above where you add the
queue and bindings via qpid-config I expect that your straw poll would
have a different outcome.
Frase
On 11/02/14 19:34, CLIVE wrote:
Ted,
Thanks for the response and your comments.
I have had to handle the case of multiple Receivers attaching to the
same queue on several occasions; primarily because the customer has
felt that it was easier to handle one queue with multiple bindings (up
to 100), rather than having a hundred queues with single bindings;
message order was also a contributing factor.
The point of the post was just to raise it as a possible issue for
future improvement..
I carried out a straw poll of 10 developers today at work. I gave them
the two examples previously described and asked what they would expect
to happen for the case where multiple Receivers were created for the
same queue. They all expected the correct Receiver to be returned from
the nextReceiver method, not the undeterministic behaviour that they
would see.
I wouldn't have thought that it would take that much code/effort to
add some additional functionality in the messaging API Implementation
to support the behavior that, it would appear, most developers would
expect to see. If I find some time I will take a look and see how it
could be done.
Clive
On 10/02/2014 22:17, Ted Ross wrote:
Clive,
What you are observing is what I expect: In the second scenario
where you use the same queue for each of the three receivers, the
receiver that receives any particular message will be non-deterministic.
This is because the binding key is applied between the exchange and
the queue (i.e. it is used to determine which queue(s) the message
should be enqueued on). Multiple receivers on a queue will receive
messages from the queue in an undetermined order, but no message
shall be delivered to more than one receiver. In the second case,
all of the messages are placed on the same queue in the order in
which they arrive. The queue acts as a buffer between the routing
rule that matched the message and the receiver that provided the
routing rule.
It would be simpler to do the following:
Rxer 1 - "amq.topic/bill; {link: {x-declare: {auto-delete:true}}}"
Re: Message API - Real world usage issue
Clive,
In your second scenario I don't see why you'd expect the message to get
delivered to Rxer 3.
What you've done is deliberately create three bindings to the same
queue, so you'd have a match bill OR ben OR tim. If you sent a message
with a subject of tim it would simply get added do the queue (that all
receivers are consuming from) so it would simply arrive at the next
available receiver.
Basically the three receivers have just been configured to receive
exactly the same thing. If you do qpid-config -r queues you'll see
queue1 with three bindings.
I think that you are confusing yourself because it might look like the
x-bindings relate to the receiver given how the Address string looks,
but as I say you're establishing bindings between amq.topic and queue1
and doing what you've done creates three ORed bindings.
I'm still struggling to see why you want to use a single queue, when it
seems like you actually want to demultiplex.
Can you explain a bit more about what you are actually trying to achieve?
Frase
On 10/02/14 21:39, CLIVE wrote:
Fraser,
Thanks for the response. The real problem is that the behavior of a
Receiver is different depending on the multiplicity of the binding
strategy used. If you use a single queue with a single binding then
messages will get delivered to the required receiver. If you use
multiple Receivers bound to the same queue, the Receiver called by the
messaging API, when delivering a message to your application, may not
be the one that you think!!
So if I create three Receivers in the same application, with the
following bindings (note unique queue names)
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue2; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue3; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And then send a message on the amq.topic exchange with a subject of
'tim'. Then Rxer3 will get returned by the 'nextReceiver' method on
the associated Session object.
But if I change the bindings so they related to the same queue
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And send the same message again, Which Receiver would you expect to
get returned from the sessions nextReceiver method?
I would expect the same result as in the first example, Rxer 3. But
this does not happen, anyone of the three receivers might get called.
This doesn't seem right to me and as a result you have to produce
quite a bit of application level logic to handle this scenario;
especially when your bindings are being passed down to you dynamically
by several client applications.
Hope this explains it a bit better than my last attempt.
Clive
On 07/02/2014 10:03, Fraser Adams wrote:
On 06/02/14 19:07, CLIVE wrote:
Hi,
[snip]
The first use case requires the dynamic creation of Receivers, but
before creating a new receiver, I would like to know if I already
have a receiver that would match the required binding. This is not
possible at the moment because the binding matching algorithms are
hidden from public view; they are buried deep inside the Brokers
Exchange Implementation code.
You know that you can get the binding information from QMF don't you
Clive? I guess I'm missing what you're looking for if it's something
different than that. And I guess to be fair to get the information
via QMF you'd need a bit of code, but I'd have thought that this
would be the most appropriate way to get the information.
Out of curiosity why do you need to know if you already have a
receiver that would match the binding?
One thing that's worth mentioning, I'm suspecting that (like me)
you've mainly been using AMQP 0.10 - If I'm reading you correctly you
sound like you are dynamically creating queue nodes and passing
x-bindings.
I've been doing that for a few years, but a few weeks back I started
looking at AMQP 1.0 and that primarily takes a perspective of
addressing the topic like exchanges and the queues end up being
subscription queues and all of the stuff that relates to binding and
the like ends up in the link (not node) config.
For me at any rate that was quite a different perspective on things
(I wrote up what I was up to in the "A write up of some AMQP 1.0
Experiments" post) previously I've been focussing on the queues, so I
was dynamica
Re: Message API - Real world usage issue
Ted,
Thanks for the response and your comments.
I have had to handle the case of multiple Receivers attaching to the
same queue on several occasions; primarily because the customer has felt
that it was easier to handle one queue with multiple bindings (up to
100), rather than having a hundred queues with single bindings; message
order was also a contributing factor.
The point of the post was just to raise it as a possible issue for
future improvement..
I carried out a straw poll of 10 developers today at work. I gave them
the two examples previously described and asked what they would expect
to happen for the case where multiple Receivers were created for the
same queue. They all expected the correct Receiver to be returned from
the nextReceiver method, not the undeterministic behaviour that they
would see.
I wouldn't have thought that it would take that much code/effort to add
some additional functionality in the messaging API Implementation to
support the behavior that, it would appear, most developers would expect
to see. If I find some time I will take a look and see how it could be done.
Clive
On 10/02/2014 22:17, Ted Ross wrote:
Clive,
What you are observing is what I expect: In the second scenario where
you use the same queue for each of the three receivers, the receiver
that receives any particular message will be non-deterministic.
This is because the binding key is applied between the exchange and
the queue (i.e. it is used to determine which queue(s) the message
should be enqueued on). Multiple receivers on a queue will receive
messages from the queue in an undetermined order, but no message shall
be delivered to more than one receiver. In the second case, all of
the messages are placed on the same queue in the order in which they
arrive. The queue acts as a buffer between the routing rule that
matched the message and the receiver that provided the routing rule.
It would be simpler to do the following:
Rxer 1 - "amq.topic/bill; {link: {x-declare: {auto-delete:true}}}"
Rxer 2 - "amq.topic/ben; {link: {x-declare: {auto-delete:true}}}"
Rxer 3 - "amq.topic/tim; {link: {x-declare: {auto-delete:true}}}"
This will give you the determinism you want. This will cause the
creation of a temporary queue for each receiver that will receive the
messages that match the topic key (following the slash in the address).
-Ted
On 02/10/2014 04:39 PM, CLIVE wrote:
Fraser,
Thanks for the response. The real problem is that the behavior of a
Receiver is different depending on the multiplicity of the binding
strategy used. If you use a single queue with a single binding then
messages will get delivered to the required receiver. If you use
multiple Receivers bound to the same queue, the Receiver called by
the messaging API, when delivering a message to your application, may
not be the one that you think!!
So if I create three Receivers in the same application, with the
following bindings (note unique queue names)
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue2; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue3; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And then send a message on the amq.topic exchange with a subject of
'tim'. Then Rxer3 will get returned by the 'nextReceiver' method on
the associated Session object.
But if I change the bindings so they related to the same queue
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And send the same message again, Which Receiver would you expect to
get returned from the sessions nextReceiver method?
I would expect the same result as in the first example, Rxer 3. But
this does not happen, anyone of the three receivers might get called.
This doesn't seem right to me and as a result you have to produce
quite a bit of application level logic to handle this scenario;
especially when your bindings are being passed down to you
dynamically by several client applications.
Hope this explains it a bit better than my last attempt.
Clive
On 07/02/2014 10:03, Fraser Adams wrote:
On 06/02/14 19:07, CLIVE wrote:
Hi,
[snip]
The first use case requires the dynamic creation of Receivers, but
before creating a new receiver, I would like to know if I already
have a receiver that would match the required bind
Re: Message API - Real world usage issue
Clive,
What you are observing is what I expect: In the second scenario where
you use the same queue for each of the three receivers, the receiver
that receives any particular message will be non-deterministic.
This is because the binding key is applied between the exchange and the
queue (i.e. it is used to determine which queue(s) the message should be
enqueued on). Multiple receivers on a queue will receive messages from
the queue in an undetermined order, but no message shall be delivered to
more than one receiver. In the second case, all of the messages are
placed on the same queue in the order in which they arrive. The queue
acts as a buffer between the routing rule that matched the message and
the receiver that provided the routing rule.
It would be simpler to do the following:
Rxer 1 - "amq.topic/bill; {link: {x-declare: {auto-delete:true}}}"
Rxer 2 - "amq.topic/ben; {link: {x-declare: {auto-delete:true}}}"
Rxer 3 - "amq.topic/tim; {link: {x-declare: {auto-delete:true}}}"
This will give you the determinism you want. This will cause the
creation of a temporary queue for each receiver that will receive the
messages that match the topic key (following the slash in the address).
-Ted
On 02/10/2014 04:39 PM, CLIVE wrote:
Fraser,
Thanks for the response. The real problem is that the behavior of a
Receiver is different depending on the multiplicity of the binding
strategy used. If you use a single queue with a single binding then
messages will get delivered to the required receiver. If you use
multiple Receivers bound to the same queue, the Receiver called by the
messaging API, when delivering a message to your application, may not
be the one that you think!!
So if I create three Receivers in the same application, with the
following bindings (note unique queue names)
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue2; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue3; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And then send a message on the amq.topic exchange with a subject of
'tim'. Then Rxer3 will get returned by the 'nextReceiver' method on
the associated Session object.
But if I change the bindings so they related to the same queue
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And send the same message again, Which Receiver would you expect to
get returned from the sessions nextReceiver method?
I would expect the same result as in the first example, Rxer 3. But
this does not happen, anyone of the three receivers might get called.
This doesn't seem right to me and as a result you have to produce
quite a bit of application level logic to handle this scenario;
especially when your bindings are being passed down to you dynamically
by several client applications.
Hope this explains it a bit better than my last attempt.
Clive
On 07/02/2014 10:03, Fraser Adams wrote:
On 06/02/14 19:07, CLIVE wrote:
Hi,
[snip]
The first use case requires the dynamic creation of Receivers, but
before creating a new receiver, I would like to know if I already
have a receiver that would match the required binding. This is not
possible at the moment because the binding matching algorithms are
hidden from public view; they are buried deep inside the Brokers
Exchange Implementation code.
You know that you can get the binding information from QMF don't you
Clive? I guess I'm missing what you're looking for if it's something
different than that. And I guess to be fair to get the information
via QMF you'd need a bit of code, but I'd have thought that this
would be the most appropriate way to get the information.
Out of curiosity why do you need to know if you already have a
receiver that would match the binding?
One thing that's worth mentioning, I'm suspecting that (like me)
you've mainly been using AMQP 0.10 - If I'm reading you correctly you
sound like you are dynamically creating queue nodes and passing
x-bindings.
I've been doing that for a few years, but a few weeks back I started
looking at AMQP 1.0 and that primarily takes a perspective of
addressing the topic like exchanges and the queues end up being
subscription queues and all of the stuff that relates to binding and
the like ends up in the link (not node) config.
For me at any rate that
Re: Message API - Real world usage issue
Fraser,
Thanks for the response. The real problem is that the behavior of a
Receiver is different depending on the multiplicity of the binding
strategy used. If you use a single queue with a single binding then
messages will get delivered to the required receiver. If you use
multiple Receivers bound to the same queue, the Receiver called by the
messaging API, when delivering a message to your application, may not be
the one that you think!!
So if I create three Receivers in the same application, with the
following bindings (note unique queue names)
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue2; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue3; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And then send a message on the amq.topic exchange with a subject of
'tim'. Then Rxer3 will get returned by the 'nextReceiver' method on the
associated Session object.
But if I change the bindings so they related to the same queue
Rxer 1 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'bill'}]}}"
Rxer 2 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'ben'}]}}"
Rxer 3 - "queue1; {create: receiver, node:
{x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
queue: 'queue1', key: 'tim'}]}}"
And send the same message again, Which Receiver would you expect to get
returned from the sessions nextReceiver method?
I would expect the same result as in the first example, Rxer 3. But this
does not happen, anyone of the three receivers might get called.
This doesn't seem right to me and as a result you have to produce quite
a bit of application level logic to handle this scenario; especially
when your bindings are being passed down to you dynamically by several
client applications.
Hope this explains it a bit better than my last attempt.
Clive
On 07/02/2014 10:03, Fraser Adams wrote:
On 06/02/14 19:07, CLIVE wrote:
Hi,
[snip]
The first use case requires the dynamic creation of Receivers, but
before creating a new receiver, I would like to know if I already
have a receiver that would match the required binding. This is not
possible at the moment because the binding matching algorithms are
hidden from public view; they are buried deep inside the Brokers
Exchange Implementation code.
You know that you can get the binding information from QMF don't you
Clive? I guess I'm missing what you're looking for if it's something
different than that. And I guess to be fair to get the information via
QMF you'd need a bit of code, but I'd have thought that this would be
the most appropriate way to get the information.
Out of curiosity why do you need to know if you already have a
receiver that would match the binding?
One thing that's worth mentioning, I'm suspecting that (like me)
you've mainly been using AMQP 0.10 - If I'm reading you correctly you
sound like you are dynamically creating queue nodes and passing
x-bindings.
I've been doing that for a few years, but a few weeks back I started
looking at AMQP 1.0 and that primarily takes a perspective of
addressing the topic like exchanges and the queues end up being
subscription queues and all of the stuff that relates to binding and
the like ends up in the link (not node) config.
For me at any rate that was quite a different perspective on things (I
wrote up what I was up to in the "A write up of some AMQP 1.0
Experiments" post) previously I've been focussing on the queues, so I
was dynamically creating queue nodes and passing x-bindings in AMQP
0.10, but in AMQP 1.0 I've been addressing the exchanges (topic type
nodes) and using the link to specify what I need. For me it took a bit
of getting used to because I was so ingrained doing it the other way,
but I think I'm getting it now.
The second use case in question requires a client application to
dynamically create multiple receivers for the same queue, but with
slightly different binding keys bound to an exchange. When a message
from an exchange gets put in the queue and delivered to the client
(via a receiver)
I'm not sure if I'm correctly interpreting what you are saying here,
so you want a client that has a single queue, but each receiver adds
different binding keys right? You do know that this will result in
what amounts to an OR condition - both keys will be bound and a
message will be put on the queue if either match so consumer A of the
queue would receive messages due to consumer B's key - is that what
you mean.
The following AMQP 1.0 consumers will do what you seem to be
Re: Message API - Real world usage issue
On 06/02/14 19:07, CLIVE wrote:
Hi,
[snip]
The first use case requires the dynamic creation of Receivers, but
before creating a new receiver, I would like to know if I already have
a receiver that would match the required binding. This is not possible
at the moment because the binding matching algorithms are hidden from
public view; they are buried deep inside the Brokers Exchange
Implementation code.
You know that you can get the binding information from QMF don't you
Clive? I guess I'm missing what you're looking for if it's something
different than that. And I guess to be fair to get the information via
QMF you'd need a bit of code, but I'd have thought that this would be
the most appropriate way to get the information.
Out of curiosity why do you need to know if you already have a receiver
that would match the binding?
One thing that's worth mentioning, I'm suspecting that (like me) you've
mainly been using AMQP 0.10 - If I'm reading you correctly you sound
like you are dynamically creating queue nodes and passing x-bindings.
I've been doing that for a few years, but a few weeks back I started
looking at AMQP 1.0 and that primarily takes a perspective of addressing
the topic like exchanges and the queues end up being subscription queues
and all of the stuff that relates to binding and the like ends up in the
link (not node) config.
For me at any rate that was quite a different perspective on things (I
wrote up what I was up to in the "A write up of some AMQP 1.0
Experiments" post) previously I've been focussing on the queues, so I
was dynamically creating queue nodes and passing x-bindings in AMQP
0.10, but in AMQP 1.0 I've been addressing the exchanges (topic type
nodes) and using the link to specify what I need. For me it took a bit
of getting used to because I was so ingrained doing it the other way,
but I think I'm getting it now.
The second use case in question requires a client application to
dynamically create multiple receivers for the same queue, but with
slightly different binding keys bound to an exchange. When a message
from an exchange gets put in the queue and delivered to the client
(via a receiver)
I'm not sure if I'm correctly interpreting what you are saying here, so
you want a client that has a single queue, but each receiver adds
different binding keys right? You do know that this will result in what
amounts to an OR condition - both keys will be bound and a message will
be put on the queue if either match so consumer A of the queue would
receive messages due to consumer B's key - is that what you mean.
The following AMQP 1.0 consumers will do what you seem to be saying,
there's a single shared subscription queue called queue1, the first
consumer binds *.news the second *.weather
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.topic/*.news; {node: {capabilities: [shared]}, link: {name: queue1}}"
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.topic/*.weather; {node: {capabilities: [shared]}, link: {name:
queue1}}"
qpid-config -r queues gives
Queue 'queue1'
bind [queue1] => ''
bind [*.news] => amq.topic
bind [*.weather] => amq.topic
For AMQP 0.10 the following would create a similar effect (not sure if
you want auto delete or not, if not remove the x-declare below and for
the AMQP 1.0 example above add reliability: at-least-once to the link Map)
./drain -b localhost -f \
"queue1; {create: receiver, node: {x-declare:{auto-delete:True},
x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: '*.news'}]}}"
./drain -b localhost -f \
"queue1; {create: receiver, node: {x-declare:{auto-delete:True},
x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: '*.weather'}]}}"
The following also works for AMQP 0.10
./drain -b localhost -f \
"queue1; {create: receiver, node: {x-declare:{auto-delete:True}}, link:
{x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: '*.news'}]}}"
./drain -b localhost -f \
"queue1; {create: receiver, node: {x-declare:{auto-delete:True}}, link:
{x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: '*.weather'}]}}"
Don't know if this is what you are looking for.
Note that in none of the cases above have I worked out how to remove a
binding other than by removing the queue so if you add the first then
the second then delete the second both bindings remain in place - I did
wonder about putting the x-declare/auto delete stuff on the link in the
second AMQP 0.10 example, but that doesn't seem to remove the binding,
so I'm not sure if that's possible.
I need to route the message to the correct application level
destination(s). To do this I need to undertake a matching operation
between the routing key of the message and the binding key(s) of the
created receivers; qpid does not deliver the message to the receiver
with the most exact binding key match.
I guess than I'm not understanding you here. As far as
