Thanks again for another update Gordon. Some responses inline below.
On 03/02/14 21:08, Gordon Sim wrote:
On 02/03/2014 07:53 PM, Fraser Adams wrote:
On 03/02/14 15:38, Gordon Sim wrote:
They should be set for outgoing messages as of 0.24[1]. What version
were you using?
I was actually using 0.27 off trunk built a couple of weeks ago.
Which properties didn't work for you using the x-amqp-* name?
I think that we may be speaking at cross purposes on this one, when you
said "They should be set for outgoing messages as of 0.24[1]. What
version were you using? " I had assumed that you were referring to the
comments around the "to" being observed by the receiver if sent from
AMQP 0.10 but not from AMQP 1.0 and the comment around that happening in
the AMQP 0.10 -> AMQP 1.0 code. Re "Which properties didn't work for you
using the x-amqp-* name" I'm not aware of any issue there (though I've
not tried many) certainly it works fine for x-amqp-to as in the example:
./spout --connection-options {protocol:amqp1.0} -b localhost -P
x-amqp-to=amqp://localhost/amq.fanout --content "Hello World" "amq.fanout"
[...]
The 0-10 to 1.0 conversion currently maps the 0-10 message-transfer's
'destination' onto to the 'to' field. At the time I did it it seemed
logical enough, but I'm happy to remove that if it is confusing (as it
probably is).
It's a difficult call, I personally suspect that it's potentially
confusing. When I was trying this stuff out I wasn't sure if it was
something that AMQP 0.10 qpid::messaging did or was an artefact of the
mapping. It's probably safest to expect it to be explicitly added like
it is on Messenger, might be worth a wider discussion though.
In translating from 0-10 to 1.0 it can't really be explicitly added,
unless you mean as a special entry in the 0-10 application-properties?
(On 1.0 the default is indeed to require iut to be explicitly set).
I think that you've misinterpreted what I was meaning by "explicitly
added" here, what I was simply meaning was expecting x-amqp-to to be set
by the producer. If as you say that on 1.0 the default is to require it
to be explicitly set the perhaps it's not unreasonable to expect
applications (even if still using 0.10 on the producer side) to actually
set it (x-amqp-to will just look like an ordinary property to 0.10 won't
it).
I'm kind of easy on this I guess as long as the pure 1.0 to 1.0 behaves
in a consistent manner when compared to say Messenger then it's probably
a coin toss as to what to do when translating.
[...]
I added the incoming and outgoing link entities as they seemed to be
useful when thinking in terms of the 1.0 model. The source and target
are as they appear in the attach received.
Yeah those seem to have potential, I stumbled across the "domain" and
"incoming"/"outgoing" stuff I've not had a chance to play yet but the
ability to establish links with arbitrary AMQP 1.0 containers seems
really cool. Is this stuff ultimately intended to replace the
"traditional" link/bridge federation model?
Listing the incoming and outgoing entities is useful even for a
standalone broker. Creating them offers a mechanism similar to the
links/bridges in 0-10.
Yeah I can see why listing the incoming/outgoing entities would be
useful for a standalone broker - presumably being able to navigate from
a subscription queue to a link would enable me to see say the
filters/selectors and other properties of the link (I'd be really nice
to be able to see what filters/selectors were present).
But my question was more about technical direction, as I say for AMQP
1.0 systems is this ultimately intended to become the primary way to
enable federation? TBH from what I've seen it looks quite neat, but I'll
need to play a bit more to compare and contrast that approach with
"traditional" federation.
* Reply-to
The qpid::messaging library doesn't particularly restrict the value of
the reply-to in the message itself[4].
If it can be interpreted as of the form <name>/<subject>, then the
address object returned will have both the name and subject set to the
respective substrings. Otherwise the name will contain the full
reply-to.
If the application takes that address and uses it directly as an
address for creating a sender, the name is used as the source or
target address for the link and if a subject is specified it will be
used to set a filter.
I believe a common usage will have reply-to simply contain a node name
(often for a temporary queue). That pattern, essentially the
client-server example, works against several brokers and also the
dispatch router[5]. It is also familiar from JMS.
However if some other specific scheme/format is used, the application
would responsible for interpreting the address and e.g. establishing
separate connections etc if that behaviour is desired.
I'll need to have more of a play with Reply-To but the thing I was
really trying to get at was that given the peer to peer model of AMQP
1.0 it is (presumably) possible to set a Reply-To to the complete unique
address of the sender (including host:port part) and the receiver could
presumably directly address the sender without necessarily returning via
intermediaries. That scenario seemed awkward for a connection oriented
paradigm such as qpid::messaging or JMS because the host:port part might
be an entirely different connection to the one it has used to connect to
say a broker.
With qpid::messaging you can set a reply-to of almost any form on a
message as well as getting it off a received message.
However passing the reply-to directly back to createReceiver() will
result in an attach being sent for a receiver link with the source set
to the node name in the address. It can't sensibly do anything else.
If you want to use other schemes, such as something that identifies a
different process to connect to or whatever, then the application
needs to handle the reply to according to whatever scheme is in use.
So if I'm interpreting you correctly here; if say I'm a sender on
localhost on port 5673 I might want to set a fully qualified reply to
AMQP address of say:
amqp://localhost:5673/response
Now a consumer could in theory happily send a response directly to that
address because of the peer to peer nature of AMQP 1.0, but I think what
you are saying is that in qpid::messaging if such a ReplyTo was send
there is no automagic mechanism given such an Address that if you pass
it to createReceiver() it will cleverly create a Connection/Session.
Similarly then I guess that if you got such an address on a
|*getJMSReplyTo
<http://docs.oracle.com/javaee/5/api/javax/jms/Message.html#getJMSReplyTo%28%29>*()|
you wouldn't simply be able to do |*send
<http://docs.oracle.com/javaee/5/api/javax/jms/MessageProducer.html#send%28javax.jms.Destination,%20javax.jms.Message%29>*(Destination
<http://docs.oracle.com/javaee/5/api/javax/jms/Destination.html> destination,
Message
<http://docs.oracle.com/javaee/5/api/javax/jms/Message.html> message)|
on a MessageProducer. So what you'd have to do would be to parse the
ReplyTo to see if it had an amqp://<host>:<port> part and if it did
create the necessary connection/session stuff and use the node name part
for the actual qpid::messaging Address.
So that makes some level of sense, but that's what I was meaning when I
said that this scenario (which seems quite reasonable given the peer to
peer nature of AMQP 1.0) was slightly *awkward* for connection oriented
APIs. Presumably if say Proton Messenger was the consumer you'd simply
be able to take the ReplyTo in this case and use that as as the address
when you do pn_message_set_address(message, address); So I think that
there is some scope for confusion/complexity as to *exactly* what an
address means as the interpretation is subtly different between
connection oriented and message oriented paradigms.
I guess that if you're always using node names on the container that the
consumer is connected to then everything is OK, but if not isn't there
scope for things getting a bit interesting - surely a consumer is
decoupled from a producer so it doesn't necessarily know how it has set
its reply to (as a node or a fully qualified address).
[...]
* Shared subscriptions
The 'shared' subscription capability is described in the AMQP 1.0
readme.
Yeah I did notice it there, my problem was working out what the syntax I
needed to use was.
Note that it is a qpidd specific extension. At present the
subscription queue will be autodeleted unless you set the link to be
durable or reliable.
Ah I didn't realise that the reliable flag would help me. So would that
be something like reliability:|||at-least-once| in the link properties?
Yes
Just to confirm that doing:
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.fanout; {node: {capabilities: [shared]}, link: {name: test-link,
reliability: at-least-once}}"
Does indeed set autoDelete to false on the subscription queue.
However that raises another question in my mind (sorry for being a pain
in the butt!!!)
In AMQP 0.10 as far as I was aware the default reliability was actually
at-least-once, or at any rate I don't believe that the default was
unreliable. The reason I'm thinking this was because I always had to
explicitly do link: {reliability: unreliable} for a couple of use cases
where I didn't want my consumers to have to acknowledge the messages.
This stuff above *seems* to point to the default reliability in AMQP 1.0
as being unreliable - is that observation correct, if so there's clearly
an inconsistency between AMQP 0.10 and 1.0
So are you saying that other AMQP 1.0 brokers generally don't support
shared subscriptions? TBH that seems unfortunate I know that I find them
incredibly useful to distribute workload between physical consumers.
They do likely support them, but they either won't expose that
capability over AMQP 1.0 yet or they will do so in a different way (I
haven't seen any other approach documented or I would have attempted
to follow it).
Just because it's not documented doesn't mean it doesn't exist ;-p
If that were the case........
Supporting an expiry policy of 'never' should be possible as well.
I'll update the test to note the convention around queue name.
What I was getting at in that section of my post was "I couldn't see
anything in the qpid::messaging code related to expiry" in other words
there was no reference to pn_terminus_set_expiry_policy so (I think)
it's additional code that's needed - have I missed something?
Right, there is no support as yet for controlling the expiration
policy (right now link scope is assumed). Adding 'never' should not be
difficult.
Cool, I guess expiry policy is probably the most *obvious* way do do
this from the spec (well to me anyway, which probably doesn't count for
much).
[...]
So that's slightly worrying I hadn't clocked that. What I'm not clear
about then is how I'd go about mapping the scenario I had with:
x-bindings: [{exchange: 'amq.match', queue: 'testqueue', key: 'key1',
arguments: {x-match: all, data-service: amqp-delivery, item-owner:
fadams}}, {exchange: 'amq.match', queue: 'testqueue', key: 'key2',
arguments: {x-match: all, data-service: http-delivery, item-owner:
fadams}}]
In this AMQP 0.10 case I've got two separate bindings added between
amq.match and "testqueue" so messages will be delivered if either of
them match.
I (naively) thought that multiple filters was the way (though on
reflection I can see that all filters must match - how else would my
topic plus selector example work the way it did d'Oh).
Can you think of a way how the use case I've suggested might be
supported (aside from selectors - they are cool, but it'd be good to be
able to replicate the legacy headers stuff completely - especially for
migration/integration purposes).
Not really, I guess a new filter type of some kind would be one path.
However I do think that using selectors is better and should work well
even for migration/integration. You could have a headers filter that
matched everything (e.g. just amq.match:any) and then specify the rest
of the matching logic as a selector and your receiver would behave the
same as if it had two headers bindings.
I was thinking about this. So I agree that ultimately using selectors is
better (and I did do exactly what you say here in my "Bringing it all
together" example albeit being even more fancy with the addition of
topic filtering too). So don't get me wrong I really do like message
selectors, but I also think that the legacy headers stuff really ought
to be able to support everything the current AMQP 0.10 model can,
especially as it's *nearly* there. So at the moment the headers stuff
with a single x-bindings value looks like this.
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.match; {link: {name: test-link, filter: {name: key, descriptor:
'apache.org:legacy-amqp-headers-binding:map', value: {x-match: all,
data-service: amqp-delivery, item-owner: fadams}}}}"
Looking in AddressHelper.cpp from what I can make out filters get added
in AddressHelper::configure and in there there's a block that iterates
through the filters and for each calls write(filter, i->value);
Unless I'm mistaken that value is a Variant::
Now in the example above it is clearly a Map, but I'm wondering about
the possibility of extending what gets accepted for
apache.org:legacy-amqp-headers-binding to either:
a) Include another variant of this such as
apache.org:legacy-amqp-headers-binding:list such that you could have a
subscription like:
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.match; {link: {name: test-link, filter: {name: headers, descriptor:
'apache.org:legacy-amqp-headers-binding:list', value: [{x-match: all,
data-service: amqp-delivery, item-owner: fadams}, {x-match: all,
data-service: http-delivery, item-owner: fadams}]}}}"
In other words the value part is a list of maps. Alternatively
b) For the apache.org:legacy-amqp-headers-binding:map use an x- property
(perhaps even x-bindings) whose value is a list e.g. something like:
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.match; {link: {name: test-link, filter: {name: headers, descriptor:
'apache.org:legacy-amqp-headers-binding:map', value: {x-bindings:
[{x-match: all, data-service: amqp-delivery, item-owner: fadams},
{x-match: all, data-service: http-delivery, item-owner: fadams}]}}}}"
You could possibly even use the original x-bindings as the property of
the filter's "value" Map though the exchange and queue parts wouldn't
make sense, but it would give a handy place for the key - so, for example:
./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.match; {link: {name: test-link, filter: {name: headers, descriptor:
'apache.org:legacy-amqp-headers-binding:map', value: {x-bindings: [{key:
'key1', arguments: {x-match: all, data-service: amqp-delivery, item-owner:
fadams}}, { key: 'key2', arguments: {x-match: all, data-service:
http-delivery, item-owner:
fadams}}]}}}}"
So as far as I can see that technically doesn't change the semantic of
the filter as the type is a map, the existing approach for a single
binding should still work but by interpreting x-bindings you'd
essentially have the ability to do exactly what you can currently do,
which seems kind of cool.
I've no idea how much work this would be though - as I say, sorry for
being a pain :-)
[...]
However we could probably look at ways to (optionally?) make this more
lenient. It certainly is also something that should be highlighted
more prominently in documentation somehow.
It would be *really really good* to be able to make this more lenient
(would quoted field names be a possibility e.g.
'data-service'='amqp-delivery')?
I've not looked at the parsing code, but I would think something like
that should be possible and I agree that it would be useful since AMQP
itself places no restrictions on property names.
Pleeeeeaaaaaassssseee :-)
Seriously, finding a way to work through the hyphen issue would be awesome.
[...]
I believe the selector filters are supported by a number of brokers
outside Qpid (ActiveMQ, HornetQ, SwiftMQ).
That's useful to know, have you tried many interoperability scenarios?
I've successfully tried selectors against ActiveMQ. The only quirk is
that it doesn't actually recognise the filters descriptor, but just
recognises the filter name as used by the current qpid JMS over 1.0
client. SO the selector shorthand doesn't work, but a fully specified
filter will.
I'm not quite sure I understand what you mean here, as I understand it
"selector: " is a qpid shortand, but I *thought* that over the wire that
got translated to a filter with a descriptor
apache.org:selector-filter:string so I *thought*
selector: \"data-service='amqp-delivery' and item-owner='fadams'\"}
was shorthand for
filter: {name: selector, descriptor:
'apache.org:selector-filter:string', value:
\"data-service='amqp-delivery' and item-owner='fadams'\"}
So are you saying that it accepts
filter: {name: selector, descriptor: 'x-filter-jms-selector', value:
\"data-service='amqp-delivery' and item-owner='fadams'\"}
Or simply link : {x-filter-jms-selector: \"data-service='amqp-delivery'
and item-owner='fadams'\"}
I *thought* that in AMQP 0.10 it went something like:
{link:{x-subscribe:{arguments:{x-filter-jms-selector:\"data-service='amqp-delivery'
and item-owner='fadams'\"}}}}
But that won't work in AMQP 1.0 because the x-subscribe is forbidden.
Actually that reminds me - so in JMS I could specify selectors on the
link as with qpid::messaging, but the JMS API also allows selectors to
be specified from the Session |*createConsumer
<http://docs.oracle.com/javaee/5/api/javax/jms/Session.html#createConsumer%28javax.jms.Destination,%20java.lang.String%29>*(Destination
<http://docs.oracle.com/javaee/5/api/javax/jms/Destination.html> destination,
String
<http://java.sun.com/j2se/1.5/docs/api/java/lang/String.html> messageSelector)|
I probably shouldn't as this, but..... are you suggesting that at the
moment with Qpid JMS specifying it from the Session will result in a
different syntax on the link attach? If so will they both work with the
Qpid brokers?
Sorry for all the crazy questions, there's a lot of subtlety in all of this.
Cheers,
Frase
