Re: QMF REST API - was Re: How to use lifetime policies
Glad that I was able to help out.
I'm a little bit curious why you're choosing to go with the REST
interface directly? What language are you programming in.
Not that I've got anything against the REST API, but I mainly put it in
place to be able to talk QMF from a browser environment.
If you are programming in Java you'd probably find it easier to use the
Java QMF API directly, if you're doing python then use the API that
qpid-config etc. use, from C++ it's a little more awkward but not too
hard to send QMF2 protocol requests out (it's ultimately just a bunch of
Map Messages) I've attached a couple of examples put together by Gordon
Sim to illustrate this in C++ the queue_depth is self explanatory, the
one marked example.cpp does something similar to qpid-route - not
exactly what you want for adding a queue, but it illustrates invoking
QMF2 methods via the raw QMF2 protocol and you can use that in
conjunction with what I've shown below on the REST API (this bit
{
_method_name:create,
_arguments: {type: queue, name: davin_test2, properties:
{durable: true}}
}
)
It is quite fun doing this stuff via curl, but it's pretty hardcore :-)
I'd *probably* take easier options in Python/Java/C++/JavaScript unless
I wanted to prove a point (clearly REST is handy if you want to mash up
info from lots of different sources in a web app).
BTW in case you are curious the reason that the URI's are relatively
complex e.g.
GET http://127.0.0.1:8080/qpid/connection/default/console/objects/queue
rather than say
GET http://127.0.0.1:8080/qpid/console/objects/queue
Is because it's perfectly possible to PUT an arbitrary number of QMF
Console connections and access them independently so the GUI can talk to
multiple discrete Consoles and I can have multiple browser instances
served by the same REST API instance.
Cheers,
Frase
On 27/01/14 22:06, Shearer, Davin wrote:
Thanks Frase,
I read and followed along through your explanation and was able to browse
various objects and finally create a queue via the REST API backed by the
C++ broker using only curl (yay!). Hopefully I can use the interface to
create one dynamically with the delete on close attribute. I'll give that
a go tomorrow!
On Thu, Jan 23, 2014 at 6:54 PM, Fraser Adams fraser.ad...@blueyonder.co.uk
wrote:
Hey Davin,
Robbie has hit the nail on the head. One somewhat unfortunate thing that
you probably ought to be aware of is that the management and monitoring for
the Qpid C++ and Java Brokers got somewhat erm fragmented for various
reasons back in the mists of time.
The C++ Broker uses Qpid Management Framework (QMF), but that has had an
interesting evolution and at some point in history QMFv1 migrated to QMFv2
which caused issues for the Java Broker and they ended up going down a
different path.
I primarily use the C++ broker, but conversely my team mainly use Java, so
I started out doing a Java implementation of the QMF2 API without knowing
much of that background. There was a bit of comedy 'cause the Qpid Java
guys didn't take a lot of notice of what I was doing 'cause it was QMF and
the C++ guys didn't really notice 'cause it was Java :-D I was a bit in no
mans land really :-/
It wasn't really until I put the GUI together that anyone really noticed
and the penny dropped.
So as Robbie says the REST API that I put together is *actually* a REST
binding for QMF2 and (apart from being a REST API) doesn't bear too much
relationship to the Java Broker's REST interface (for most of history we
were largely unaware of what each other was up to).
It's a far from ideal position and Rob, Gordon, Robbie and I have chatted
a fair bit about the need to get a more cohesive view. Ultimately the right
answer will be to adopt the AMQP Management Specification across the board,
though that's likely to be some way off (I certainly need to find some time
to properly get my head around it).
By way of making a stand for the unity of Qpid Brokers :-) I put
together a QMF2 plugin for the Java Broker, it was one of those things that
one does on principle - really it was as much as anything a statement to
say really guys it *is* actually possible to provide a cohesive view. One
thing to bear in mind with the Java Broker QMF plugin is that I've tried
fairly hard to map as much as possible from the Java Broker internal
Management Model to QMF Management Objects, but there are differences in
the Management Models, so there are certainly more itty bitty things that
can be controlled by the Java Broker's native management interface than via
QMF (mainly 'cause Robbie keeps adding features ;-) but OTOH pretty much
anything you can do with qpid-config will work with both the C++ and Java
Brokers.
So to get back to the QMF REST API, the best place to look is the JavaDoc
(yes really, there is actually a ton of JavaDoc for this stuff)
start in qpid/tools/src/java/docs/api/index.html
Then look for package org.apache.qpid.restapi and class QpidServer
To get you
Re: QMF REST API - was Re: How to use lifetime policies
Re: I like the decoupling that REST provides as it just takes a library that speaks HTTP. I don't have to link against a vendor-specific management library nor do I have to carry through their dependencies. What you say about REST is indeed true and is a good selling point, but it's worth bearing in mind that it's *also* true of the approach illustrated in the examples from Gordon - that approach is using the QMF *protocol* and in terms of dependencies it is simply using pure AMQP as a transport passing Map Messages in a request/response pattern to specific AMQP Destinations, that's arguably no more coupled nor vendor specific than say using the REST API, even the Java QMF stuff isn't really coupled as it's implemented as pure JMS. In actual fact it's the QMF aspects that are vendor specific - QMF is specific to the Qpid C++ broker. I certainly don't want to dissuade the approach that you're taking, but you should probably be armed with all the facts - sure you're not dependent on a library, but as I say above nor are you really dependent on a library for some of the other approaches (though Gordon's examples are C++ using qpid::mess I like the decoupling that REST provides as it just takes a library that speaks HTTP. I don't have to link against a vendor-specific management library nor do I have to carry through their dependencies. aging) I've not tried raw QMF via Proton Messenger, it might well be possible though it currently seems to have quite a few limitations wrt. topic subscriptions (and other filter types). As I say ultimately QMF is actually your biggest vendor specific dependency, so that's something that you should keep in the back of your mind going forward. I expect QMF to be around for quite some time, but there is certainly a desire to move towards the AMQP 1.0 Management Specification over time. As we move forward on that I'm sure that there will be a concerted effort to avoid breaking existing QMF things (I'm certainly intending to try my best to avoid that) and as I say it's likely to be some time before traction builds in that direction, but it's certainly something that you should be cognoscente of if you are building a mission critical system - put a comment in your code to track AMQP 1.0 Management developments on this list at the very least. Hope that this stuff is useful, Cheers, Frase On 28/01/14 19:18, Shearer, Davin wrote: On Tue, Jan 28, 2014 at 1:39 PM, Fraser Adams fraser.ad...@blueyonder.co.uk wrote: Glad that I was able to help out. I'm a little bit curious why you're choosing to go with the REST interface directly? What language are you programming in. I am programming this piece in C (messaging and transport layer of a larger system). I am using proton for the AMQP layer and using the REST API for the broker management. I like the decoupling that REST provides as it just takes a library that speaks HTTP. I don't have to link against a vendor-specific management library nor do I have to carry through their dependencies. I don't want the consumers of the library (be it additional middle-ware layers or the application-layer code) to be concerned about the messaging and transport details, just that we moving data reliably from one end-point to another over a flexible topology. Not that I've got anything against the REST API, but I mainly put it in place to be able to talk QMF from a browser environment. It should work for my needs as well. It took longer than I'd hoped to package a reference broker, but I've managed it. I've attached a couple of examples put together by Gordon Sim to illustrate this in C++ the queue_depth is self explanatory, the one marked example.cpp does something similar to qpid-route - not exactly what you want for adding a queue, but it illustrates invoking QMF2 methods via the raw QMF2 protocol and you can use that in conjunction with what I've shown below on the REST API I think your REST API will be sufficient provided I request a dynamic queue with delete on close via REST. If that isn't possible then it could be the case that there will be abandoned reply queues lingering in the broker if a client creates one, then dies without removing it. I will review the code since more information usually leads to better decisions. It is quite fun doing this stuff via curl, but it's pretty hardcore :-) I'd *probably* take easier options in Python/Java/C++/JavaScript unless I wanted to prove a point (clearly REST is handy if you want to mash up info from lots of different sources in a web app). The commandline curl stuff is to understand the API, get my URLs right and to test the behaviour before writing it up in C using libcurl. BTW in case you are curious the reason that the URI's are relatively complex e.g. GET http://127.0.0.1:8080/qpid/connection/default/console/objects/queuerather than say GET http://127.0.0.1:8080/qpid/console/objects/queue Is because it's perfectly possible to PUT
Re: QMF REST API - was Re: How to use lifetime policies
By having this transport and messaging layer, I can swap out the broker management implementations (hopefully) without changing the API. If my customers demand that I support a different AMQP 1.0 vendor, then I can load in a different management module. If they have a REST API, then I take on no additional dependencies. If I could have managed to link libqpid* statically, it would have helped. Once the management spec arrives and there is a proton-like library implementing that spec, then I'll have my final solution. I've learned a bunch from you guys over the past couple of weeks. Thanks so much for the help!
Re: QMF REST API - was Re: How to use lifetime policies
Many thanks for the clarification Robbie, I guess that the full picture continues to offer us some warnings from history. At one level the issue Given that many of the subsequent schema enhancements then werent implemented or simply couldnt be becuase they didnt fit the Java brokers model, be it due to the virtualhost disparity, protocol differences, or lower level things like the memory management essentially still exists (albeit without a direct schema coupling). One of the things that worries me a little is that in many ways (given my far from perfect understanding) QMF2 and AMQP 1.0 Management Spec. share many similarities (no mandatory schema - it's all introspectable being encoded as AMQP Maps) that's great and offers the ability to align well at the protocol level, but the differences in the management models between the two Qpid brokers and of course with other AMQP brokers and services will continue to make developing cohesive tooling a challenge. That's life I guess, I guess that you can't expect Microsoft/IBM/RabbitQ/ActiveMQ/Qpid/etc to offer identical feature sets but AMQP 1.0 management isn't going to fix that. As you say the Java Broker has evolved at lot and you've got a lot of quite low level features in the Management Model that are exposed in the embedded Web GUI. What I'm not clear on is how far your thinking has progressed along the options for eventually externalising this via AMQP 1.0 Management and making it available on external tooling. That's *largely* what I've done in the QMF2 space - albeit incomplete/imperfect/whatever, as I say I think that there are a lot of similarities between QMF2 and AMQP 1.0 Management and a big part of my motivation was proving that it was possible and exposing the sorts of wrinkles that we might expect (in particular I guess that I've only exposed the lowest common denominator). Perhaps the generality of AMQP 1.0 and the needs of Vendors to make their product sets readily differentiable might mean that we'll never hit the Nirvana of one Vendor's tooling managing another Vendor's broker - I mentioned to Rob back at Easter that I worried that the lack of schemas might make this a lot harder too. I wouldn't want to be coupled to a schema like QMF1 is, but having a schema available might make it easier to create generic tooling that is *semantically* meaningful. Without it the most we can expect from a generic tool is to be a bit like a JMX Console, which is OK but a bit ugly. Thanks again for the background, it's really interesting and I think knowledge of the history is incredibly useful to help us (try to) avoid the same wrinkles going forward. Frase On 24/01/14 01:11, Robbie Gemmell wrote: On 23 January 2014 23:54, Fraser Adams fraser.ad...@blueyonder.co.ukwrote: Hey Davin, Robbie has hit the nail on the head. One somewhat unfortunate thing that you probably ought to be aware of is that the management and monitoring for the Qpid C++ and Java Brokers got somewhat erm fragmented for various reasons back in the mists of time. Just to clarify things a little since much of this was before your (and to a smaller extent, even my own) time with the project, it was probably less that they became fragmented and more that they started that way. The Java broker initially had JMX based management functionality (as is/was popular for Java things to do) satisfying its model, and as the C++ broker developed it obviously needed managmement tooling to go with it, but didnt have the luxury of JMX built right into its runtime platform...though did have a nice AMQP interface sitting there, and so QMF grew to fill that need and beyond. Thus on the managemnet side things essentially started out different and then grew from there. In early 2010 we did add some initial support for QMF1 to the Java broker with aim of moving them closer together, but while we were still working on building that out the development on QMF itself moved away from an update of its existing binary protocol and instead toward creating what became the largely-new QMF2, making for a semi-start-over scenario. We didn't get round to that and the incomplete QMF1 support also withered over time, causing pain here and there due to continued enhancement of the C++ broker and the management schema then leading to breakages in the Java build. Given that many of the subsequent schema enhancements then werent implemented or simply couldnt be becuase they didnt fit the Java brokers model, be it due to the virtualhost disparity, protocol differences, or lower level things like the memory management, and in the end we removed the QMF1 support (or more accurately didnt bringing it along with us when we would have had to completely overhaul it to do so) at the time we begun to modularize the brokers management support out to plugins and started working on overhauling the configuration model toward its current form. Part of that change was us turning thoughts
Re: How to use lifetime policies
I've got the rest API up and attached to my broker. I can log into it
using my browser and even create new queues using the web app. What I
can't seem to use is curl for doing REST interactions. For instance, I
fire up the C++ broker on 127.0.0.1 on port 5673, then startup the REST API
using the default authentication:
[davin@laptop24 qpid]$ ./build/tools/src/java/bin/QpidRestAPI.sh -a
127.0.0.1:5673 -w ./build/tools/src/java/bin/qpid-web
log4j:WARN No appenders could be found for logger
(org.apache.qpid.restapi.QpidServer).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
I create a queue called 'davin_test via the (very nice) web app and then
verify it via qpid-config:
[davin@laptop24 qpid]$ qpid-config -a 127.0.0.1:5673 queues
Queue Name Attributes
==
2a8ce7c5-52cb-4958-b472-e7439d02b0d7:0.0 auto-del excl
TempQueue568cf2e2-7bf9-4472-ab86-cb11bc7f52ad auto-del excl --argument
no-local=False
TempQueue73e4df4c-9ef5-4d1e-ab6c-c8ba1647365a auto-del excl --argument
no-local=False
TempQueue750b42a4-118c-4be4-9f6e-38818de00c3e auto-del excl --argument
no-local=False
davin_test
[davin@laptop24 qpid]$
It's there, great, now lets get some details using curl and the REST API:
[davin@laptop24 qpid]$ curl -X GET -u admin:admin
http://127.0.0.1:8080/rest/queue/davin_test
404 Not Found: File /rest/queue/davin_test not found.[davin@laptop24 qpid]$
Darn, that didn't work... can we put new queues?
[davin@laptop24 qpid]$ curl -X PUT -u admin:admin -d '{durable:true}'
http://127.0.0.1:8080/rest/queue/default/davin_test2
405 Bad Method.[davin@laptop24 qpid]$
What am I missing?
Re: How to use lifetime policies
On 23 January 2014 19:27, Shearer, Davin dshea...@novetta.com wrote:
I've got the rest API up and attached to my broker. I can log into it
using my browser and even create new queues using the web app. What I
can't seem to use is curl for doing REST interactions. For instance, I
fire up the C++ broker on 127.0.0.1 on port 5673, then startup the REST API
using the default authentication:
[davin@laptop24 qpid]$ ./build/tools/src/java/bin/QpidRestAPI.sh -a
127.0.0.1:5673 -w ./build/tools/src/java/bin/qpid-web
log4j:WARN No appenders could be found for logger
(org.apache.qpid.restapi.QpidServer).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
I create a queue called 'davin_test via the (very nice) web app and then
verify it via qpid-config:
[davin@laptop24 qpid]$ qpid-config -a 127.0.0.1:5673 queues
Queue Name Attributes
==
2a8ce7c5-52cb-4958-b472-e7439d02b0d7:0.0 auto-del excl
TempQueue568cf2e2-7bf9-4472-ab86-cb11bc7f52ad auto-del excl --argument
no-local=False
TempQueue73e4df4c-9ef5-4d1e-ab6c-c8ba1647365a auto-del excl --argument
no-local=False
TempQueue750b42a4-118c-4be4-9f6e-38818de00c3e auto-del excl --argument
no-local=False
davin_test
[davin@laptop24 qpid]$
It's there, great, now lets get some details using curl and the REST API:
[davin@laptop24 qpid]$ curl -X GET -u admin:admin
http://127.0.0.1:8080/rest/queue/davin_test
404 Not Found: File /rest/queue/davin_test not found.[davin@laptop24qpid]$
Darn, that didn't work... can we put new queues?
[davin@laptop24 qpid]$ curl -X PUT -u admin:admin -d '{durable:true}'
http://127.0.0.1:8080/rest/queue/default/davin_test2
405 Bad Method.[davin@laptop24 qpid]$
What am I missing?
Those URLs look a lot like ones that would be used against the REST API
from the Java brokers management-http plugin. The URLs/payloads used by the
'QMF REST API' from the tools Fraser put together aren't likely to be that
similar to them.
I'll now have to defer to someone who can tell you what they would be
like...
Robbie
Re: How to use lifetime policies
Ah, OK, I thought they'd both use the same REST API as documented here http://qpid.apache.org/releases/qpid-0.24/java-broker/book/Java-Broker-Configuring-And-Managing-HTTP-Management.html. To me the language that the broker is written in is an implementation detail. Would it not make sense to layer the same REST API over both the C++ and Java brokers? Why should it be different?
Re: How to use lifetime policies
On 23 January 2014 22:25, Shearer, Davin dshea...@novetta.com wrote: Ah, OK, I thought they'd both use the same REST API as documented here http://qpid.apache.org/releases/qpid-0.24/java-broker/book/Java-Broker-Configuring-And-Managing-HTTP-Management.html . Alas no, as per the URL those are the docs for the Java broker HTTP management plugin. To me the language that the broker is written in is an implementation detail. The brokers came into existence separately and have evolved differently over time. While effort is often made (from Gordon and Rob in particular) to ensure they support similar features in similar ways where possible, they ultimately do differ significantly and probably most visibly so around configuration and management. Would it not make sense to layer the same REST API over both the C++ and Java brokers? Why should it be different? Frasers QMF2 tools can work against either to an extent (it was originally developed against the C++ broker and then a QMF2 plugin added to allow operation against the Java broker), but the brokers are different and so some stuff just is different. As one small example: one of your URLs had a virtualhost name in it, but the C++ broker doesnt use named/multiple virtualhosts and thus that part just doesnt map particularly nicely between the two. I guess the main reason the two REST APis in question are different though is because the Java brokers plugin was developed to be an HTTP interface more directly exposing its configuration model (because almost everything is now configurable via it), whereas the QMF REST API was developed as a way of offering up the QMF functionality used by the C++ broker for its management. The two things were basically developed by different people at different times for somewhat different purposes. Robbie
Re: How to use lifetime policies
Thanks for filling me in! Is there any documentation on the C++ REST API? On Thu, Jan 23, 2014 at 6:12 PM, Robbie Gemmell robbie.gemm...@gmail.comwrote: On 23 January 2014 22:25, Shearer, Davin dshea...@novetta.com wrote: Ah, OK, I thought they'd both use the same REST API as documented here http://qpid.apache.org/releases/qpid-0.24/java-broker/book/Java-Broker-Configuring-And-Managing-HTTP-Management.html . Alas no, as per the URL those are the docs for the Java broker HTTP management plugin. To me the language that the broker is written in is an implementation detail. The brokers came into existence separately and have evolved differently over time. While effort is often made (from Gordon and Rob in particular) to ensure they support similar features in similar ways where possible, they ultimately do differ significantly and probably most visibly so around configuration and management. Would it not make sense to layer the same REST API over both the C++ and Java brokers? Why should it be different? Frasers QMF2 tools can work against either to an extent (it was originally developed against the C++ broker and then a QMF2 plugin added to allow operation against the Java broker), but the brokers are different and so some stuff just is different. As one small example: one of your URLs had a virtualhost name in it, but the C++ broker doesnt use named/multiple virtualhosts and thus that part just doesnt map particularly nicely between the two. I guess the main reason the two REST APis in question are different though is because the Java brokers plugin was developed to be an HTTP interface more directly exposing its configuration model (because almost everything is now configurable via it), whereas the QMF REST API was developed as a way of offering up the QMF functionality used by the C++ broker for its management. The two things were basically developed by different people at different times for somewhat different purposes. Robbie -- Davin Shearer Engineer 8830 Stanford Blvd, Suite 306 Columbia, MD 21045 443-741-4517
QMF REST API - was Re: How to use lifetime policies
Hey Davin,
Robbie has hit the nail on the head. One somewhat unfortunate thing that
you probably ought to be aware of is that the management and monitoring
for the Qpid C++ and Java Brokers got somewhat erm fragmented for
various reasons back in the mists of time.
The C++ Broker uses Qpid Management Framework (QMF), but that has had an
interesting evolution and at some point in history QMFv1 migrated to
QMFv2 which caused issues for the Java Broker and they ended up going
down a different path.
I primarily use the C++ broker, but conversely my team mainly use Java,
so I started out doing a Java implementation of the QMF2 API without
knowing much of that background. There was a bit of comedy 'cause the
Qpid Java guys didn't take a lot of notice of what I was doing 'cause it
was QMF and the C++ guys didn't really notice 'cause it was Java :-D I
was a bit in no mans land really :-/
It wasn't really until I put the GUI together that anyone really noticed
and the penny dropped.
So as Robbie says the REST API that I put together is *actually* a REST
binding for QMF2 and (apart from being a REST API) doesn't bear too much
relationship to the Java Broker's REST interface (for most of history we
were largely unaware of what each other was up to).
It's a far from ideal position and Rob, Gordon, Robbie and I have
chatted a fair bit about the need to get a more cohesive view.
Ultimately the right answer will be to adopt the AMQP Management
Specification across the board, though that's likely to be some way off
(I certainly need to find some time to properly get my head around it).
By way of making a stand for the unity of Qpid Brokers :-) I put
together a QMF2 plugin for the Java Broker, it was one of those things
that one does on principle - really it was as much as anything a
statement to say really guys it *is* actually possible to provide a
cohesive view. One thing to bear in mind with the Java Broker QMF
plugin is that I've tried fairly hard to map as much as possible from
the Java Broker internal Management Model to QMF Management Objects, but
there are differences in the Management Models, so there are certainly
more itty bitty things that can be controlled by the Java Broker's
native management interface than via QMF (mainly 'cause Robbie keeps
adding features ;-) but OTOH pretty much anything you can do with
qpid-config will work with both the C++ and Java Brokers.
So to get back to the QMF REST API, the best place to look is the
JavaDoc (yes really, there is actually a ton of JavaDoc for this stuff)
start in qpid/tools/src/java/docs/api/index.html
Then look for package org.apache.qpid.restapi and class QpidServer
To get you started if you do (assuming the same host and port as below):
http://127.0.0.1:8080/qpid/connection/default/console/objects/queue
You will retrieve a list of all of the QMF Queue Management Objects in
the form of a JSON list (you can do it in a browser too)
http://127.0.0.1:8080/qpid/connection/default/console/objects/exchange
will retrieve the exchanges
http://127.0.0.1:8080/qpid/connection/default/console/objects/connection
will retrieve the connections
and so on.
the default bit is a convenience, so this is a true REST API that
mirrors the QMF API pretty closely so you really ought to first PUT a
QMF Console Connection before you do anything else, but of course PUT is
a pain from a browser, so if you specify default for the connection
part of the URI the API transparently creates (or uses an already
created) default QMF Console instance.
The GUI actually has a JavaScript class that provides a JavaScript
implementation of the QMF2 API, which is backed by the REST API, so the
GUI is generally abstracted from the REST API, there's a factory method
for that which does the HTTP PUT under the hood
You can also use POST in order to invoke QMF methods (for
adding/deleting queues etc.).
To be honest I'm a bit rusty - I tend to use the GUI or at worse the
JavaScript wrapper which abstracts the detail,
and I've not done it the hard way for the best part of 15 months, so
the following is pulled out from the code.
The POST syntax is:
POST: host:port/qpid/connection/name/object/ObjectId
HTTP body: {_method_name:method,_arguments:inArgs}
method: A string containing the QMF2 method name e.g. getLogLevel, setLogLevel,
create, delete.
inArgs: A JSON string containing the method arguments e.g.
{level:debug+:Broker} for setLogLevel.
HTTP response: A JSON string containing the response e.g.
{level:notice+} for getLogLevel (may be empty).
This method invokes the QMF2 method method with arguments inArgs on the
object ObjectId
And it works like QMF, so what you'd first need to do is to recover the Broker
Management Object
http://127.0.0.1:8080/qpid/connection/default/console/objects/broker
You need to do this because the majority of the QMF CRUD methods are applied on
the Broker Management Object,
Re: How to use lifetime policies
As one small example: one of your URLs had a virtualhost name in it, but the C++ broker doesnt use named/multiple virtualhosts and thus that part just doesnt map particularly nicely between the two. Robbie FWIW I did actually make an attempt to do something useful wrt. virtualhosts (though nicely might not be accurate, as you suggest...). The QMF model does technically have a virtualhost part, but none of the tooling actually honours it. If I was only going to make the QMF plugin for the Java Broker work for the GUI I suspect I might have been able to use that, but I was quite keen to be able to use qpid-config too because there have been a few postings on the user list about the availability of command line tools like qpid-config for the Java Broker (also, as you know, at least in part one of my motivations was to show in kickable terms that convergence was at least plausible and to have a platform to be able to illustrate the wrinkles that make convergence exciting). I've somewhat kludged it so if there's a non default virtualhost the queue/exchange names get prefixed e.g. vhost:/my-queue The prefixing happens in the plugin and the :/ is a convention that doesn't seem too unreasonable, you can also do things like: qpid-config add queue vhost:/another-queue And the plugin notices the :/ and splits things into the vhost and queue parts so will add the queue another-queue to the virtualhost thus named. It's not nice for sure, but it does mean that it works for a non-modded qpid-config/qpid-queue-stat etc.. Frase - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: QMF REST API - was Re: How to use lifetime policies
On 23 January 2014 23:54, Fraser Adams fraser.ad...@blueyonder.co.ukwrote: Hey Davin, Robbie has hit the nail on the head. One somewhat unfortunate thing that you probably ought to be aware of is that the management and monitoring for the Qpid C++ and Java Brokers got somewhat erm fragmented for various reasons back in the mists of time. Just to clarify things a little since much of this was before your (and to a smaller extent, even my own) time with the project, it was probably less that they became fragmented and more that they started that way. The Java broker initially had JMX based management functionality (as is/was popular for Java things to do) satisfying its model, and as the C++ broker developed it obviously needed managmement tooling to go with it, but didnt have the luxury of JMX built right into its runtime platform...though did have a nice AMQP interface sitting there, and so QMF grew to fill that need and beyond. Thus on the managemnet side things essentially started out different and then grew from there. In early 2010 we did add some initial support for QMF1 to the Java broker with aim of moving them closer together, but while we were still working on building that out the development on QMF itself moved away from an update of its existing binary protocol and instead toward creating what became the largely-new QMF2, making for a semi-start-over scenario. We didn't get round to that and the incomplete QMF1 support also withered over time, causing pain here and there due to continued enhancement of the C++ broker and the management schema then leading to breakages in the Java build. Given that many of the subsequent schema enhancements then werent implemented or simply couldnt be becuase they didnt fit the Java brokers model, be it due to the virtualhost disparity, protocol differences, or lower level things like the memory management, and in the end we removed the QMF1 support (or more accurately didnt bringing it along with us when we would have had to completely overhaul it to do so) at the time we begun to modularize the brokers management support out to plugins and started working on overhauling the configuration model toward its current form. Part of that change was us turning thoughts towards an embedded web management GUI to facilitate removing the Eclipse RCP based management console that existed previously, which then lead us to adding a REST API to power it (as has become a more popular way of managing things) which simply got based directly around the brokers new configuration model...and that about sums up how things ended up as they are now. As you later mention, we will be looking to add AMQP Management based functionality in time as that specification progresses. The C++ Broker uses Qpid Management Framework (QMF), but that has had an interesting evolution and at some point in history QMFv1 migrated to QMFv2 which caused issues for the Java Broker and they ended up going down a different path. I primarily use the C++ broker, but conversely my team mainly use Java, so I started out doing a Java implementation of the QMF2 API without knowing much of that background. There was a bit of comedy 'cause the Qpid Java guys didn't take a lot of notice of what I was doing 'cause it was QMF and the C++ guys didn't really notice 'cause it was Java :-D I was a bit in no mans land really :-/ It wasn't really until I put the GUI together that anyone really noticed and the penny dropped. So as Robbie says the REST API that I put together is *actually* a REST binding for QMF2 and (apart from being a REST API) doesn't bear too much relationship to the Java Broker's REST interface (for most of history we were largely unaware of what each other was up to). It's a far from ideal position and Rob, Gordon, Robbie and I have chatted a fair bit about the need to get a more cohesive view. Ultimately the right answer will be to adopt the AMQP Management Specification across the board, though that's likely to be some way off (I certainly need to find some time to properly get my head around it). By way of making a stand for the unity of Qpid Brokers :-) I put together a QMF2 plugin for the Java Broker, it was one of those things that one does on principle - really it was as much as anything a statement to say really guys it *is* actually possible to provide a cohesive view. One thing to bear in mind with the Java Broker QMF plugin is that I've tried fairly hard to map as much as possible from the Java Broker internal Management Model to QMF Management Objects, but there are differences in the Management Models, so there are certainly more itty bitty things that can be controlled by the Java Broker's native management interface than via QMF (mainly 'cause Robbie keeps adding features ;-) but OTOH pretty much anything you can do with qpid-config will work with both the C++ and Java Brokers. So to get back to the QMF REST API, the best place to look is
Re: How to use lifetime policies
Any chance for this patch to make it into the upcoming 0.26 release?
request for 0.26 (was Re: How to use lifetime policies)
On 01/20/2014 03:09 PM, Shearer, Davin wrote: Any chance for this patch to make it into the upcoming 0.26 release? That is largely down to the release manager, and its getting late in the cycle. Justin, the JIRA in question is https://issues.apache.org/jira/browse/QPID-5463. Chuck and Alan both had a look at the patch and approve its inclusion in 0.26 if it is not too late. --Gordon. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: request for 0.26 (was Re: How to use lifetime policies)
I know it's late in the cycle and I just realized this AM that the fix was not included, but it would be really helpful to me if it were included. On Mon, Jan 20, 2014 at 10:52 AM, Gordon Sim g...@redhat.com wrote: On 01/20/2014 03:09 PM, Shearer, Davin wrote: Any chance for this patch to make it into the upcoming 0.26 release? That is largely down to the release manager, and its getting late in the cycle. Justin, the JIRA in question is https://issues.apache.org/ jira/browse/QPID-5463. Chuck and Alan both had a look at the patch and approve its inclusion in 0.26 if it is not too late. --Gordon. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org -- Davin Shearer
Re: request for 0.26 (was Re: How to use lifetime policies)
I appreciate it immensely, thanks guys! On Mon, Jan 20, 2014 at 4:23 PM, Gordon Sim g...@redhat.com wrote: On 01/20/2014 05:13 PM, Shearer, Davin wrote: I know it's late in the cycle and I just realized this AM that the fix was not included, but it would be really helpful to me if it were included. It's on the 0.26 release branch now. Thanks Justin, Alan and Chuck! On Mon, Jan 20, 2014 at 10:52 AM, Gordon Sim g...@redhat.com wrote: On 01/20/2014 03:09 PM, Shearer, Davin wrote: Any chance for this patch to make it into the upcoming 0.26 release? That is largely down to the release manager, and its getting late in the cycle. Justin, the JIRA in question is https://issues.apache.org/ jira/browse/QPID-5463. Chuck and Alan both had a look at the patch and approve its inclusion in 0.26 if it is not too late. --Gordon. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org -- Davin Shearer Engineer 8830 Stanford Blvd, Suite 306 Columbia, MD 21045 443-741-4517
Re: How to use lifetime policies
On 01/17/2014 02:34 AM, Shearer, Davin wrote: Hi Frase, So still no dice. What jar file has javax.jms.* (like javax.jms.Connection)? geronimo-jms_1.1_spec-1.0.jar I put qpid-all.jar in my CLASSPATH, failed, then explicitly added all the jar files in qpid/src/java/build/lib: export CLASSPATH=$(for j in /home/davin/external/qpid/src/java/build/lib/*.jar; do echo -n $j:;done) and failed again. The java broker I was using was from the latest release archive. It was already built for me. You need the *client* release, or else you can use the build directory from a source build (which includes both client and broker). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
Hey Gordon and Davin, I think Gordon's comment around the client release is very pertinent, though checking through an earlier mail from Davin he says: I'm very interested in the REST API for the C++ broker and I've got git-svn-id:https://svn.apache.org/repos/asf/qpid/trunk@155854913f79535-47bb-0310-9956-ffa450edef68 checked out. I went into the qpid/java directory and ran: Looking through that in a browser that does contain the Java Broker and Client stuff as far as I can see, so doing ant in the qpid/java/ directory *should* build both. My current best guess is still that the QPID_HOME environment variable hasn't been set, though clearly it'll make sense for Davin to actually check which jars have been build and put into qpid/src/java/build/lib Frase On 17/01/14 09:49, Gordon Sim wrote: On 01/17/2014 02:34 AM, Shearer, Davin wrote: Hi Frase, So still no dice. What jar file has javax.jms.* (like javax.jms.Connection)? geronimo-jms_1.1_spec-1.0.jar I put qpid-all.jar in my CLASSPATH, failed, then explicitly added all the jar files in qpid/src/java/build/lib: export CLASSPATH=$(for j in /home/davin/external/qpid/src/java/build/lib/*.jar; do echo -n $j:;done) and failed again. The java broker I was using was from the latest release archive. It was already built for me. You need the *client* release, or else you can use the build directory from a source build (which includes both client and broker). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
Happy to report that I've built the java tools. Frase nailed it. QPID_HOME was set wrong. I had it set at the top level qpid directory instead of src/java/build. So all I needed to build was to set: QPID_HOME=/home/davin/external/qpid/src/java/build CLASSPATH=/home/davin/external/qpid/src/java/build/lib/qpid-all.jar Then pop into the tools/src/java directory and run 'ant all'. Thanks!
Re: How to use lifetime policies
Great news!! Have fun with it :-) Frase On 17/01/14 15:25, Shearer, Davin wrote: Happy to report that I've built the java tools. Frase nailed it. QPID_HOME was set wrong. I had it set at the top level qpid directory instead of src/java/build. So all I needed to build was to set: QPID_HOME=/home/davin/external/qpid/src/java/build CLASSPATH=/home/davin/external/qpid/src/java/build/lib/qpid-all.jar Then pop into the tools/src/java directory and run 'ant all'. Thanks! - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
On 01/15/2014 10:22 PM, Shearer, Davin wrote: Hi Frase, I'm very interested in the REST API for the C++ broker and I've got git-svn-id: https://svn.apache.org/repos/asf/qpid/trunk@155854913f79535-47bb-0310-9956-ffa450edef68 checked out. I went into the qpid/java directory and ran: ant -f build.xml build BUILD SUCCESSFUL Cool, so then I went into the tools/src/java directory and ran: ant all ...and I get a ton of errors all with 'error: package javax.jms does not exist'. I'm obviously missing a pre-requisite. I'm on Fedora19. What am I missing? Probably the qpid JMS client... the quick fix is to add qpid-all.jar to your classpath from a client release or from a build from trunk (in java/build/lib). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
Hi Davin, My *guess* (hope!!!) is that it's an environment/classpath thing When I do an svn update of qpid trunk I generally do: cd qpid-root/java ant Which builds the Qpid Java stuff (and look like your first step, though I just do ant not ant -f build.xml build - I doubt that makes any difference though) I then do: cd qpid-root/tools/src/java ant cleanall ant all Again pretty similar to what you've done. The error does seem indicative of a classpath issue, I think that I mentioned in the README that the Qpid client jars were needed on the classpath. My .bashrc has the following stuff: ## HOME DIRECTORIES # # Set up the location of the JDK JAVA_HOME=/usr/lib/jvm/java-6-openjdk # Clearly you'd need to set this to your own path QPID_HOME=/home/fadams/qpid/qpid-trunk/qpid/java/build #QPID_WORK is the working directory for the Java Broker - again you'd need to set this to your own path QPID_WORK=/home/fadams/qpid-work PATH=$JAVA_HOME/bin:$PATH:$QPID_HOME/bin # CLASSPATH # # Set up CLASSPATH CLASSPATH=$CLASSPATH:$JAVA_HOME/lib/:$QPID_HOME/lib/qpid-all.jar:. export JAVA_HOME QPID_HOME QPID_WORK PATH CLASSPATH # End of .bashrc stuff The qpid-all.jar is mostly just a manifest jar that pulls in everything, there's probably a better way to do this, but do date for me it's simple and convenient and how it's set on my dev environment. The only thing that slightly troubles me is that from what I've gathered you've been running the Java Broker successfully, though it might be that the qpid-server startup script does some magic to work out the classpath for itself. If your problem turns out *not* to be a classpath issue then I'm slightly stumped :-( HTH, Frase On 15/01/14 22:22, Shearer, Davin wrote: Hi Frase, I'm very interested in the REST API for the C++ broker and I've got git-svn-id: https://svn.apache.org/repos/asf/qpid/trunk@155854913f79535-47bb-0310-9956-ffa450edef68 checked out. I went into the qpid/java directory and ran: ant -f build.xml build BUILD SUCCESSFUL Cool, so then I went into the tools/src/java directory and ran: ant all ...and I get a ton of errors all with 'error: package javax.jms does not exist'. I'm obviously missing a pre-requisite. I'm on Fedora19. What am I missing? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
Hi Frase, So still no dice. What jar file has javax.jms.* (like javax.jms.Connection)? I put qpid-all.jar in my CLASSPATH, failed, then explicitly added all the jar files in qpid/src/java/build/lib: export CLASSPATH=$(for j in /home/davin/external/qpid/src/java/build/lib/*.jar; do echo -n $j:;done) and failed again. The java broker I was using was from the latest release archive. It was already built for me.
Re: How to use lifetime policies
Hi Davin,
I'm afraid that you've got me stumped now :-( as I say all I've ever
needed was to have qpid-all.jar on my classpath, so basically the
environment variables I mentioned yesterday, building the java client
library stuff then ant all.
You *did* set up everything that I mentioned yesterday didn't you?
qpid root/qpid/tools/src/java/build.xml makes use of the QPID_HOME
environment variable.
!--
=== --
!-- Note that this requires the QPID_HOME environment
variable --
!-- to be set and it installs the plugin to
$QPID_HOME/lib/plugins --
!--
=== --
property environment=env/
property name=qpid.home location=${env.QPID_HOME}/
This is kind of a hangover from before this work got committed to the
core qpid code base and could probably use relative paths now, but ISTR
a few core Java thing things need (or at least used to need) QPID_HOME
so it seemed a useful path.
My current best guess is that you've just set CLASSPATH but haven't set
QPID_HOME?
If it's not that then I'm really scratching my head, if you've been able
to get any JMS clients running then it should work - ultimately it's
just a JMS Client.
Frase
On 17/01/14 02:34, Shearer, Davin wrote:
Hi Frase,
So still no dice. What jar file has javax.jms.* (like
javax.jms.Connection)? I put qpid-all.jar in my CLASSPATH, failed, then
explicitly added all the jar files in qpid/src/java/build/lib:
export CLASSPATH=$(for j in
/home/davin/external/qpid/src/java/build/lib/*.jar; do echo -n $j:;done)
and failed again.
The java broker I was using was from the latest release archive. It was
already built for me.
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
Re: # node name and other addressing stuff - was Re: How to use lifetime policies
On 01/10/2014 03:30 PM, Fraser Adams wrote: Cheers Rob, glad to know I wasn't *totally* losing my marbles and this stuff really *is* a black art :'( Well glad in the loosest sense, totally agree with your last sentence! On 10/01/14 14:55, Rob Godfrey wrote: So, definitely seems like we (Qpid) could use some more documentation then There is an AMQP_1.0 readme in the cpp directory. That describes the mapping of the API to AMQP 1.0 including the use of '#'. I agree it is not in the most obvious place or the most useful format yet. Part of the problem I find is that it is quite hard to modify the documentation we do have. This isn't just the process required, though that feels more complicated than it needs to be also. Its also the structure of the existing documents. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
Hi Frase, I'm very interested in the REST API for the C++ broker and I've got git-svn-id: https://svn.apache.org/repos/asf/qpid/trunk@155854913f79535-47bb-0310-9956-ffa450edef68 checked out. I went into the qpid/java directory and ran: ant -f build.xml build BUILD SUCCESSFUL Cool, so then I went into the tools/src/java directory and ran: ant all ...and I get a ton of errors all with 'error: package javax.jms does not exist'. I'm obviously missing a pre-requisite. I'm on Fedora19. What am I missing?
Re: How to use lifetime policies
On 11/01/14 02:03, Davin Shearer wrote: Wow, I didn't know about this but I'll definitely try it out. Hi again Davin, I'm afraid that I just realised that there could probably do with being some slightly better documentation. I've updated the README.txt on trunk - you can see it here: https://svn.apache.org/repos/asf/qpid/trunk/qpid/tools/src/java/README.txt I *think* that 0.24 was the first release that the Java based QMF2 stuff made it into, but it's pretty self-contained (it's really just a JMS Application) so if you are using an older release you can pull https://svn.apache.org/repos/asf/qpid/trunk/qpid/tools/src/java/ And it should work with brokers from 0.8 onwards (though there are a few quirks with 0.8, everything from 0.10 should behave properly though). Hopefully the updated README.txt is enough to get you started. Frase - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
I doubt that I can give any sort of definitive answers here as my experience of AMQP 1.0 is barely off the n00b level yet, but RE: What I need to do is create a reply queue. The AMQP 1.0 spec covers creating Dynamic Reply-To. The problem I'm having is in garbage collection of the abandoned queues as these queues have a permanent lifetime policy. If I were using qpid (rather than proton) I could simply have the client create the queue using a name of its choice with the DeleteOnClose lifetime policy and I also set exclusive to true and I get exactly what I need there This sounds like what you are actually looking for is to be able to set your queue to be exclusive and auto-delete, which is certainly something that you can do with qpid::messaging and JMS (at least for AMQP 0.10). I'd be surprised if you couldn't do this for AMQP 1.0, but there seems to be rather a lot of differences between the two and I'm still waiting for a set of AMQP 1.0 for AMQP 0.10 users documents to be written by someone who knows (hint hint :-)) My guess is that Gordon Sim might be able to help you out if he's reading this, it is Gordon who has written the AMQP 1.0 support for qpid::messaging and I believe that this is based on the Proton engine API so it's pretty likely that if setting the policies you require is possible then Gordon will know how to do it - though that said there's no guarantee that if it's possible to do with Proton engine that it'll be also supported on the Proton messenger API, but I'd be surprised if not (though messenger abstracts connection/session concepts so there are definitely conceptual differences). I'm much more familiar with qpid::messaging/JMS myself and I'd be quite keen for some guidance on AMQP 1.0 Addressing and examples of say how to write something in messenger that could publish or subscribe to things that I'm familiar with - most of my current use cases publish to amq.match - the default headers exchange and subscribe by using x-bindings. Over time I'll migrate to message selectors, but in the short term I'd love to know how to go about getting this stuff interoperable. As Robbie mentioned I've done a fair bit of work on QMF, though it's worth pointing out that the underlying implementation is built on JMS and currently uses the AMQP 0.10 client, I'll need to give things a bit of a shake through when I get more familiar with AMQP 1.0, but it's worth mentioning that the longer term journey should really be to enable AMQP 1.0 Management support - again something that I need to do when I'm a bit more familiar and have a bit of time. The Java Broker QMF plugin that I put together (take a look in qpid/tools/src/java) is pretty decent, though one of the quirks is that there are a number of differences in the underlying management models between the C++ and Java brokers. I've mapped a fair amount to cover off the most common use cases but for Java Broker specific things there are likely to be gaps. If you are using Java there are examples in qpid/tools/src/java/src/main/java/org/apache/qpid/qmf2/tools the QpidConfig example shows how to add and remove queues. The QMF stuff responds to the QMF map message protocol, so you can also use the Python qpid-config to add/delete queues (and you could lift code from that if you're programming in Python) for C++ you'd have to resort to writing map messages directly. I'm a bit rusty on that as I've tended to use the API but I could probably dig out some examples that Gordon put together if you are interested. That said in your mail you say It seems that I can create queues with the DeleteOn* lifetime policies using REST, but then the queues are immediately deleted. I guess it doesn't seem to make sense to me to try and create a queue with an auto delete policy with either QMF or the Java Broker REST API because auto delete means that the queue gets deleted after the last connection to it goes away, so but then the queues are immediately deleted would be what I'd expect. Given my last paragraph I still reckon your best bet is to hope Gordon (or Rafael Schloming) can suggest how to establish auto delete/exclusive behaviour in Proton - given my comments about such things getting zapped after the last connection goes I'd be surprised if it's not something that can be applied as part of the queue Node creation. Regards, Frase On 09/01/14 19:46, Shearer, Davin wrote: Hello, Is there a way, using proton and QMF to create a queue that has a lifetime policy other than permanent? If so, how? I am trying to build a transport library in C using proton with support for a qpid broker. What I need to do is create a reply queue. The AMQP 1.0 spec covers creating Dynamic Reply-To. The problem I'm having is in garbage collection of the abandoned queues as these queues have a permanent lifetime policy. If I were using qpid (rather than proton) I could simply have the client create the queue
Re: How to use lifetime policies
On 01/09/2014 07:46 PM, Shearer, Davin wrote: I am trying to build a transport library in C using proton with support for a qpid broker. What I need to do is create a reply queue. The AMQP 1.0 spec covers creating Dynamic Reply-To. The problem I'm having is in garbage collection of the abandoned queues as these queues have a permanent lifetime policy. If I were using qpid (rather than proton) I could simply have the client create the queue using a name of its choice with the DeleteOnClose lifetime policy and I also set exclusive to true and I get exactly what I need, but it this ties me to qpid. You can create a temporary queue when using the qpid::messaging client by simply specifying '#' as the node name in the address when creating the receiver or sender. You can get the peer-assigned address (for use in reply-to) via the getAddress() method on the Sender or Receiver instance that is returned. This uses completely standard AMQP 1.0 underneath - it sets the dynamic flag on the source/target and by default will request the delete-on-close lifetime policy (the others can be requested via node-properties in the address). Again, this is standard AMQP 1.0, nothing proprietary. It works with the qpid brokers but also e.g. ActiveMQ, ApolloMQ, HornetQ and should work with any broker that supports the dynamic flag. Using proton messenger there is at present no way to set the dynamic flag or control the source/target beyond the address. With ApolloMQ you can create a policy that will be applied if the name of the node matches, and that policy can specify the queue should be automatically deleted. The qpid c++ broker supports something every similar in the upcoming 0.26 (using the AMQP 1.0 lifetime-policy options). This allows the desired behaviour to be configured broker side. With RabbitMQ you could creating a receiving link from /exchange/amq.direct/unique-name (where unique name is e.g. a client specified UUID). That address can be given as the reply to, and when the receiver exits there will be no queue left. Unfortunately this trick doesn't work with the qpid brokers, which at present require the binding to be specified as a filter. So unfortunately at present I think it is not possible to do what you want using the java broker and proton messenger. If you can use qpid::messaging (which doesn't tie you to qpid brokers), that should work. If you can't, then you can request changes to proton messenger or - better in my view - request other brokers to support the policy driven approach from ApolloMQ (and now qpid c++!). (If I have got the wrong end of the stick and you are using the proton *engine* API, then it is indeed possible. You just set the dynamic flag on the terminus for the link. I can point you to the code in qpid::messaging that does this if it is of interest). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
On 01/10/2014 09:15 AM, Fraser Adams wrote:
I'm much more familiar with qpid::messaging/JMS myself and I'd be quite
keen for some guidance on AMQP 1.0 Addressing and examples of say how to
write something in messenger that could publish or subscribe to things
that I'm familiar with - most of my current use cases publish to
amq.match - the default headers exchange and subscribe by using
x-bindings. Over time I'll migrate to message selectors, but in the
short term I'd love to know how to go about getting this stuff
interoperable.
At present messenger only sets the address of the source for receiving
links and of the target for sending links. For sending/receiving to an
existing queue or to a fanout exchange, this is all you need (just
specify the queue or exchange name as the path in the url of the address
used, e.g. amqp://127.0.0.1/my-queue or amqp://127.0.0.1/amq.fanout).
That gives you the basic queue and topic patterns of JMS.
Binding a subscription to the headers exchange (or indeed even to topic
or direct exchanges) with both qpid brokers requires using the 'legacy
amqp' filters registered as extensions for 1.0. Likewise if specifying a
jms selector. At present there is no way to set a filter on a source
when using messenger however.
To migrate from using the headers exchange to using a selector while
retaining the qpid::messaging API should be fairly straightforward. You
just need to specify a 'selector' option in the link properties for your
address, and use e.g. a fanout exchange instead.
e.g. drain -f amq.fanout; {link:{selector:\colour IN ('red', 'blue')\}}
then:
spout amq.fanout -P colour=red
spout amq.fanout -P colour=yellow
etc.
That gives you the same functionality as the headers exchange in (I
think) a much simpler syntax, and works for both 1.0 (where it uses the
registered jms filters extension) or 0-10 (or mixtures of the two).
You can use another exchange type as well of course, with additional
filtering on subject distinct from the selector.
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
Thanks for the help.
A couple of things... being able to configure lifetime policies on the
broker based on the name of the node is a wonderful idea and I look forward
to that in qpid 0.26. Problem is that I need to have the solution work on
RHEL6 (which ships with qpid 0.14) if possible.
I took some python code from Apache Dispatch Router and modified it to work
with the qpid broker directly without using the dispatch router. Here's
what I have:
test.p
from proton import Messenger, Message
def main():
host = 127.0.0.1:5673
messenger = Messenger()
messenger.start()
reply_subscription = messenger.subscribe(amqp://%s/# % host)
reply_address = reply_subscription.address
print Reply address: %s % reply_address
request = Message()
request.address = reply_address
request.reply_to = reply_address
request.body = proton rocks
messenger.put(request)
messenger.send()
messenger.recv()
response = Message()
messenger.get(response)
print Response: %r % response.body
messenger.stop()
main()
This is just using proton, which is great, but the problem is that the
dynamic queues have a permanent lifetime policy as demonstrated here:
Window 1: bash$ qpidd --log-enable trace+ --auth no --port 5673 --interface
127.0.0.1
Window 2: bash$ python test.py
Reply address: amqp://
127.0.0.1:5673/9795cd4c-4d24-42f1-b369-dca989345467_receiver-xxx
Response: 'proton rocks'
bash$ qpid-config -a 127.0.0.1:5673 queues
Queue Name Attributes
==
9795cd4c-4d24-42f1-b369-dca989345467_receiver-xxx
c0f573d3-9b46-4af5-8792-989c6f48b91e:0.0 auto-del excl
bash$ python test.py
Reply address: amqp://
127.0.0.1:5673/ec940df5-4882-455a-bc14-49b2fb2c8ae1_receiver-xxx
Response: 'proton rocks'
[davin@laptop24 proton-scm]$ qpid-config -a 127.0.0.1:5673 queues
Queue Name Attributes
==
23e5aa68-aa52-4de4-b8fa-14c732f854a8:0.0 auto-del excl
9795cd4c-4d24-42f1-b369-dca989345467_receiver-xxx
ec940df5-4882-455a-bc14-49b2fb2c8ae1_receiver-xxx
If only there is a way to set the dynamic flag on the terminus for the link
at the Messenger level as I think that would add the auto-del attribute to
the queue. I think this is possible if I were using the proton Engine
rather than the Messenger
and I tried to piece together what I need to do from studying the code
found in proton.c, but the send loop in that code gave me a headache. I
think we just allow Messenger to set that dynamic flag when a subscription
is requested, I could use Messenger and be golden, so if I have a feature
request, that would be what I want. Messenger is built on Engine, Engine
has the ability to set the dynamic flag, what's needed is to expose that
feature in Messenger. It's all proton, all AMQP 1.0, what's not to like?
On Fri, Jan 10, 2014 at 8:01 AM, Gordon Sim g...@redhat.com wrote:
On 01/10/2014 09:15 AM, Fraser Adams wrote:
I'm much more familiar with qpid::messaging/JMS myself and I'd be quite
keen for some guidance on AMQP 1.0 Addressing and examples of say how to
write something in messenger that could publish or subscribe to things
that I'm familiar with - most of my current use cases publish to
amq.match - the default headers exchange and subscribe by using
x-bindings. Over time I'll migrate to message selectors, but in the
short term I'd love to know how to go about getting this stuff
interoperable.
At present messenger only sets the address of the source for receiving
links and of the target for sending links. For sending/receiving to an
existing queue or to a fanout exchange, this is all you need (just specify
the queue or exchange name as the path in the url of the address used, e.g.
amqp://127.0.0.1/my-queue or amqp://127.0.0.1/amq.fanout). That gives you
the basic queue and topic patterns of JMS.
Binding a subscription to the headers exchange (or indeed even to topic
or direct exchanges) with both qpid brokers requires using the 'legacy
amqp' filters registered as extensions for 1.0. Likewise if specifying a
jms selector. At present there is no way to set a filter on a source when
using messenger however.
To migrate from using the headers exchange to using a selector while
retaining the qpid::messaging API should be fairly straightforward. You
just need to specify a 'selector' option in the link properties for your
address, and use e.g. a fanout exchange instead.
e.g. drain -f amq.fanout; {link:{selector:\colour IN ('red', 'blue')\}}
then:
spout amq.fanout -P colour=red
spout amq.fanout -P colour=yellow
etc.
That gives you the same functionality as the headers exchange in (I think)
a much simpler syntax, and works for both 1.0 (where it uses the registered
jms filters extension) or 0-10 (or
# node name and other addressing stuff - was Re: How to use lifetime policies
On 10/01/14 10:24, Gordon Sim wrote: You can create a temporary queue when using the qpid::messaging client by simply specifying '#' as the node name in the address when creating the receiver or sender. You can get the peer-assigned address (for use in reply-to) via the getAddress() method on the Sender or Receiver instance that is returned. Out of curiosity where is the specifying '#' as the node name stuff specified? I've also noticed the use of $ in the AMQP 1.0 Management spec, and there's the use of ~ in messenger e.g. ~amqp://0.0.0.0 to specify a listen connection. I've grepped around the AMQP 1.0 spec and the draft Address spec and couldn't find these - well to be fair there were a lot of # in the AMQP 1.0 spec, but that was in the ASCII art diagrams :- Much of this stuff remains a bit of a black art to me I'm afraid :-( Cheers, Frase - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: # node name and other addressing stuff - was Re: How to use lifetime policies
On 10 January 2014 15:45, Fraser Adams fraser.ad...@blueyonder.co.ukwrote: On 10/01/14 10:24, Gordon Sim wrote: You can create a temporary queue when using the qpid::messaging client by simply specifying '#' as the node name in the address when creating the receiver or sender. You can get the peer-assigned address (for use in reply-to) via the getAddress() method on the Sender or Receiver instance that is returned. Out of curiosity where is the specifying '#' as the node name stuff specified? This is not AMQP 1.0, but something that is supported by the qpid::messaging API I've also noticed the use of $ in the AMQP 1.0 Management spec, Mainly this was just to give us a fair shot at not clashing with anyone's existing node names... I think it is likely that any node names defined in the AMQP specs will start amqp$, but that isn't standardized. and there's the use of ~ in messenger e.g. ~amqp://0.0.0.0 to specify a listen connection. This is Messenger specific and not part of the AMQP standards. I've grepped around the AMQP 1.0 spec and the draft Address spec and couldn't find these - well to be fair there were a lot of # in the AMQP 1.0 spec, but that was in the ASCII art diagrams :- Much of this stuff remains a bit of a black art to me I'm afraid :-( So, definitely seems like we (Qpid) could use some more documentation then :-) -- Rob Cheers, Frase - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
If I'm subscribing to a node named # doesn't it make sense that the
terminus be marked dynamic automagically?
On Fri, Jan 10, 2014 at 9:26 AM, Shearer, Davin dshea...@novetta.comwrote:
Thanks for the help.
A couple of things... being able to configure lifetime policies on the
broker based on the name of the node is a wonderful idea and I look forward
to that in qpid 0.26. Problem is that I need to have the solution work on
RHEL6 (which ships with qpid 0.14) if possible.
I took some python code from Apache Dispatch Router and modified it to
work with the qpid broker directly without using the dispatch router.
Here's what I have:
test.p
from proton import Messenger, Message
def main():
host = 127.0.0.1:5673
messenger = Messenger()
messenger.start()
reply_subscription = messenger.subscribe(amqp://%s/# % host)
reply_address = reply_subscription.address
print Reply address: %s % reply_address
request = Message()
request.address = reply_address
request.reply_to = reply_address
request.body = proton rocks
messenger.put(request)
messenger.send()
messenger.recv()
response = Message()
messenger.get(response)
print Response: %r % response.body
messenger.stop()
main()
This is just using proton, which is great, but the problem is that the
dynamic queues have a permanent lifetime policy as demonstrated here:
Window 1: bash$ qpidd --log-enable trace+ --auth no --port 5673
--interface 127.0.0.1
Window 2: bash$ python test.py
Reply address: amqp://
127.0.0.1:5673/9795cd4c-4d24-42f1-b369-dca989345467_receiver-xxx
Response: 'proton rocks'
bash$ qpid-config -a 127.0.0.1:5673 queues
Queue Name Attributes
==
9795cd4c-4d24-42f1-b369-dca989345467_receiver-xxx
c0f573d3-9b46-4af5-8792-989c6f48b91e:0.0 auto-del excl
bash$ python test.py
Reply address: amqp://
127.0.0.1:5673/ec940df5-4882-455a-bc14-49b2fb2c8ae1_receiver-xxx
Response: 'proton rocks'
[davin@laptop24 proton-scm]$ qpid-config -a 127.0.0.1:5673 queues
Queue Name Attributes
==
23e5aa68-aa52-4de4-b8fa-14c732f854a8:0.0 auto-del excl
9795cd4c-4d24-42f1-b369-dca989345467_receiver-xxx
ec940df5-4882-455a-bc14-49b2fb2c8ae1_receiver-xxx
If only there is a way to set the dynamic flag on the terminus for the
link at the Messenger level as I think that would add the auto-del
attribute to the queue. I think this is possible if I were using the
proton Engine rather than the Messenger
and I tried to piece together what I need to do from studying the code
found in proton.c, but the send loop in that code gave me a headache. I
think we just allow Messenger to set that dynamic flag when a subscription
is requested, I could use Messenger and be golden, so if I have a feature
request, that would be what I want. Messenger is built on Engine, Engine
has the ability to set the dynamic flag, what's needed is to expose that
feature in Messenger. It's all proton, all AMQP 1.0, what's not to like?
On Fri, Jan 10, 2014 at 8:01 AM, Gordon Sim g...@redhat.com wrote:
On 01/10/2014 09:15 AM, Fraser Adams wrote:
I'm much more familiar with qpid::messaging/JMS myself and I'd be quite
keen for some guidance on AMQP 1.0 Addressing and examples of say how to
write something in messenger that could publish or subscribe to things
that I'm familiar with - most of my current use cases publish to
amq.match - the default headers exchange and subscribe by using
x-bindings. Over time I'll migrate to message selectors, but in the
short term I'd love to know how to go about getting this stuff
interoperable.
At present messenger only sets the address of the source for receiving
links and of the target for sending links. For sending/receiving to an
existing queue or to a fanout exchange, this is all you need (just specify
the queue or exchange name as the path in the url of the address used, e.g.
amqp://127.0.0.1/my-queue or amqp://127.0.0.1/amq.fanout). That gives
you the basic queue and topic patterns of JMS.
Binding a subscription to the headers exchange (or indeed even to topic
or direct exchanges) with both qpid brokers requires using the 'legacy
amqp' filters registered as extensions for 1.0. Likewise if specifying a
jms selector. At present there is no way to set a filter on a source when
using messenger however.
To migrate from using the headers exchange to using a selector while
retaining the qpid::messaging API should be fairly straightforward. You
just need to specify a 'selector' option in the link properties for your
address, and use e.g. a fanout exchange instead.
e.g. drain -f amq.fanout; {link:{selector:\colour IN ('red',
'blue')\}}
then:
spout amq.fanout
Re: How to use lifetime policies
On 01/10/2014 02:26 PM, Shearer, Davin wrote: A couple of things... being able to configure lifetime policies on the broker based on the name of the node is a wonderful idea and I look forward to that in qpid 0.26. Problem is that I need to have the solution work on RHEL6 (which ships with qpid 0.14) if possible. One thing to point out is that that version of qpidd doesn't support AMQP 1.0 at all. [...] If only there is a way to set the dynamic flag on the terminus for the link at the Messenger level as I think that would add the auto-del attribute to the queue. I think this is possible if I were using the proton Engine rather than the Messenger Correct, though unless there are specific reasons not to, I would suggest using qpid::messaging rather than building your own solution on top of the engine API. and I tried to piece together what I need to do from studying the code found in proton.c, but the send loop in that code gave me a headache. I think we just allow Messenger to set that dynamic flag when a subscription is requested, I could use Messenger and be golden, so if I have a feature request, that would be what I want. A similar issue was closed with wont-fix: https://issues.apache.org/jira/browse/PROTON-426. The problem is I think that with the dynamic flag the name is actually created by the remote peer, so its not immediately obvious how this would be passed back in the messenger API. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
On 01/10/2014 03:05 PM, Shearer, Davin wrote: If I'm subscribing to a node named # doesn't it make sense that the terminus be marked dynamic automagically? That is what qpid::messaging does. It then returns the name allocated by the broker through the getAddress() method on the Receiver returned, so that you can put that in the reply-to of your message. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
On 01/10/2014 10:24 AM, Gordon Sim wrote: You can create a temporary queue when using the qpid::messaging client by simply specifying '#' as the node name in the address when creating the receiver or sender. You can get the peer-assigned address (for use in reply-to) via the getAddress() method on the Sender or Receiver instance that is returned. This uses completely standard AMQP 1.0 underneath - it sets the dynamic flag on the source/target and by default will request the delete-on-close lifetime policy (the others can be requested via node-properties in the address). Again, this is standard AMQP 1.0, nothing proprietary. It works with the qpid brokers but also e.g. ActiveMQ, ApolloMQ, HornetQ and should work with any broker that supports the dynamic flag. I should have also mentioned here that it also works against Qpid Dispatch Router. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: # node name and other addressing stuff - was Re: How to use lifetime policies
Cheers Rob, glad to know I wasn't *totally* losing my marbles and this stuff really *is* a black art :'( Well glad in the loosest sense, totally agree with your last sentence! Frase On 10/01/14 14:55, Rob Godfrey wrote: On 10 January 2014 15:45, Fraser Adams fraser.ad...@blueyonder.co.ukwrote: On 10/01/14 10:24, Gordon Sim wrote: You can create a temporary queue when using the qpid::messaging client by simply specifying '#' as the node name in the address when creating the receiver or sender. You can get the peer-assigned address (for use in reply-to) via the getAddress() method on the Sender or Receiver instance that is returned. Out of curiosity where is the specifying '#' as the node name stuff specified? This is not AMQP 1.0, but something that is supported by the qpid::messaging API I've also noticed the use of $ in the AMQP 1.0 Management spec, Mainly this was just to give us a fair shot at not clashing with anyone's existing node names... I think it is likely that any node names defined in the AMQP specs will start amqp$, but that isn't standardized. and there's the use of ~ in messenger e.g. ~amqp://0.0.0.0 to specify a listen connection. This is Messenger specific and not part of the AMQP standards. I've grepped around the AMQP 1.0 spec and the draft Address spec and couldn't find these - well to be fair there were a lot of # in the AMQP 1.0 spec, but that was in the ASCII art diagrams :- Much of this stuff remains a bit of a black art to me I'm afraid :-( So, definitely seems like we (Qpid) could use some more documentation then :-) -- Rob Cheers, Frase - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: # node name and other addressing stuff - was Re: How to use lifetime policies
On 01/10/2014 02:45 PM, Fraser Adams wrote: On 10/01/14 10:24, Gordon Sim wrote: You can create a temporary queue when using the qpid::messaging client by simply specifying '#' as the node name in the address when creating the receiver or sender. You can get the peer-assigned address (for use in reply-to) via the getAddress() method on the Sender or Receiver instance that is returned. Out of curiosity where is the specifying '#' as the node name stuff specified? That isn't sent over the wire in any way. It is just part of the qpid::messaging supported syntax. However the AMQP 1.0 it results in is completely standard. The same syntax works for 0-10 also (see e.g.the client/server example). (There are options to qpid::messaging that result in non-standard interaction where required to support some existing functionality, but this is not one of them). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
A similar issue was closed with wont-fix: https://issues.apache.org/jira/browse/PROTON-426. The problem is I think that with the dynamic flag the name is actually created by the remote peer, so its not immediately obvious how this would be passed back in the messenger API. OK, I'm missing something. In the python code using proton, I have the following lines: reply_subscription = messenger.subscribe(amqp://%s/# % host) reply_address = reply_subscription.address The reply address is being set to something like ec940df5-4882-455a-bc14-49b2fb2c8ae1_receiver-xxx. I didn't think that an AMQP 1.0 client could create nodes _that it named_ on the broker. AFAIK, proton is all AMQP 1.0 all the time (and therefore should be vendor agnostic), but yet using proton I was able to create a queue on my qpid broker. So now the question is, who named it? The spec states that the name of the dynamic queue is created by the broker (this is why the name must not be set when using dynamic). In that python code snippet above, I ask the broker for a dynamic node and I get the newly created node address via the reply_subscription.address attribute, but the qpid broker did not mark the queue as auto-del. I'd prefer to solve this using proton rather than qpid::messaging (I actually have this working using libqpid*, but proton is appealing because it is easy on its dependencies among other things). libqpidmessaging has more dependencies that I'd need to carry through when deploying my library that I'd rather not have to carry though, like boost. It is very likely that projects linking my library will be using boost and I want to avoid symbol version issues with the version of boost that they might be linking against. On Fri, Jan 10, 2014 at 10:08 AM, Gordon Sim g...@redhat.com wrote: On 01/10/2014 02:26 PM, Shearer, Davin wrote: A couple of things... being able to configure lifetime policies on the broker based on the name of the node is a wonderful idea and I look forward to that in qpid 0.26. Problem is that I need to have the solution work on RHEL6 (which ships with qpid 0.14) if possible. One thing to point out is that that version of qpidd doesn't support AMQP 1.0 at all. [...] If only there is a way to set the dynamic flag on the terminus for the link at the Messenger level as I think that would add the auto-del attribute to the queue. I think this is possible if I were using the proton Engine rather than the Messenger Correct, though unless there are specific reasons not to, I would suggest using qpid::messaging rather than building your own solution on top of the engine API. and I tried to piece together what I need to do from studying the code found in proton.c, but the send loop in that code gave me a headache. I think we just allow Messenger to set that dynamic flag when a subscription is requested, I could use Messenger and be golden, so if I have a feature request, that would be what I want. A similar issue was closed with wont-fix: https://issues.apache.org/ jira/browse/PROTON-426. The problem is I think that with the dynamic flag the name is actually created by the remote peer, so its not immediately obvious how this would be passed back in the messenger API. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org -- Davin Shearer Engineer 8830 Stanford Blvd, Suite 306 Columbia, MD 21045 443-741-4517
Re: How to use lifetime policies
I think the behaviour of qpid::messaging is correct. I don't think its out of the question to expect the same behaviour in proton messenger. On Fri, Jan 10, 2014 at 10:11 AM, Gordon Sim g...@redhat.com wrote: On 01/10/2014 03:05 PM, Shearer, Davin wrote: If I'm subscribing to a node named # doesn't it make sense that the terminus be marked dynamic automagically? That is what qpid::messaging does. It then returns the name allocated by the broker through the getAddress() method on the Receiver returned, so that you can put that in the reply-to of your message. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org -- Davin Shearer Engineer 8830 Stanford Blvd, Suite 306 Columbia, MD 21045 443-741-4517
Re: # node name and other addressing stuff - was Re: How to use lifetime policies
grepping around proton-c for '#' comes up dry. If # isn't sent over the wire, I'd expect to see it dealt with somewhere in the source code. [davin@centos6 proton-c]$ grep '#' src/*.c include/proton/*.h | grep -v ':#' On Fri, Jan 10, 2014 at 10:02 AM, Gordon Sim g...@redhat.com wrote: On 01/10/2014 02:45 PM, Fraser Adams wrote: On 10/01/14 10:24, Gordon Sim wrote: You can create a temporary queue when using the qpid::messaging client by simply specifying '#' as the node name in the address when creating the receiver or sender. You can get the peer-assigned address (for use in reply-to) via the getAddress() method on the Sender or Receiver instance that is returned. Out of curiosity where is the specifying '#' as the node name stuff specified? That isn't sent over the wire in any way. It is just part of the qpid::messaging supported syntax. However the AMQP 1.0 it results in is completely standard. The same syntax works for 0-10 also (see e.g.the client/server example). (There are options to qpid::messaging that result in non-standard interaction where required to support some existing functionality, but this is not one of them). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org -- Davin Shearer Engineer 8830 Stanford Blvd, Suite 306 Columbia, MD 21045 443-741-4517
Re: How to use lifetime policies
On 01/10/2014 04:03 PM, Shearer, Davin wrote: OK, I'm missing something. In the python code using proton, I have the following lines: reply_subscription = messenger.subscribe(amqp://%s/# % host) reply_address = reply_subscription.address The reply address is being set to something like ec940df5-4882-455a-bc14-49b2fb2c8ae1_receiver-xxx. I didn't think that an AMQP 1.0 client could create nodes _that it named_ on the broker. AFAIK, proton is all AMQP 1.0 all the time (and therefore should be vendor agnostic), but yet using proton I was able to create a queue on my qpid broker. So now the question is, who named it? The spec states that the name of the dynamic queue is created by the broker (this is why the name must not be set when using dynamic). In that python code snippet above, I ask the broker for a dynamic node and I get the newly created node address via the reply_subscription.address attribute, but the qpid broker did not mark the queue as auto-del. Many, many apologies! It appears this *has* been implemented in proton, I was just unaware of it. In the case above the dynamic flag is indeed set and the node name is the one assigned by the broker. So that should work against other 1.0 brokers also. Sorry for the misinformation folks! Messenger *can* create temporary queues. (In my defense, the status of https://issues.apache.org/jira/browse/PROTON-426 was misleading) - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
On 01/10/2014 02:26 PM, Shearer, Davin wrote: I took some python code from Apache Dispatch Router and modified it to work with the qpid broker directly without using the dispatch router. [...] This is just using proton, which is great, but the problem is that the dynamic queues have a permanent lifetime policy as demonstrated here: Ok, this now is something that could be fixed in the broker. It could (and arguably should) default to delete-on-close for dynamic nodes. I've raised an issue to track that and will fix it before too long I hope: https://issues.apache.org/jira/browse/QPID-5463 Your initial email suggested you were using the Qpid Java broker? That may default as expected (Rob?). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
CORRECTION (was Re: How to use lifetime policies)
On 01/10/2014 10:24 AM, Gordon Sim wrote: Using proton messenger there is at present no way to set the dynamic flag The above statement is incorrect. The dynamic flag can be set by specifying '#' as the path in the url to which you subscribe, e.g. amqp://127.0.0.1/# The address allocated by the peer is then communicated back via the subscription returned. I apologise again for the misinformation first time around. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
I actually _prefer_ the C++ broker. I was using the java one for the web management. So to get this straight... proton sends '#' to the qpid broker, which in turn generates a node name which it then sends back to the client via the address attribute. The only thing at is missing is that the qpid broker does not set auto-del on the newly created dynamic queue. If/When that's fixed, I'm golden. Thanks so much! On Fri, Jan 10, 2014 at 11:51 AM, Gordon Sim g...@redhat.com wrote: On 01/10/2014 02:26 PM, Shearer, Davin wrote: I took some python code from Apache Dispatch Router and modified it to work with the qpid broker directly without using the dispatch router. [...] This is just using proton, which is great, but the problem is that the dynamic queues have a permanent lifetime policy as demonstrated here: Ok, this now is something that could be fixed in the broker. It could (and arguably should) default to delete-on-close for dynamic nodes. I've raised an issue to track that and will fix it before too long I hope: https://issues.apache.org/jira/browse/QPID-5463 Your initial email suggested you were using the Qpid Java broker? That may default as expected (Rob?). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org -- Davin Shearer Engineer 8830 Stanford Blvd, Suite 306 Columbia, MD 21045 443-741-4517
Re: How to use lifetime policies
On 01/10/2014 01:34 PM, Shearer, Davin wrote: I actually _prefer_ the C++ broker. I was using the java one for the web management. So to get this straight... proton sends '#' to the qpid broker, which in turn generates a node name which it then sends back to the client via the address attribute. Not quite... The '#' is not part of the protocol between the client and broker. What actually happens is that proton (or qpid.messaging) sees the '#' in the address and instead of sending a name across to the broker, sets the dynamic flag in the attach frame, leaving the name blank. The broker sees the dynamic flag, creates a temporary queue, and sends the name of the queue back in the resulting attach frame. The only thing at is missing is that the qpid broker does not set auto-del on the newly created dynamic queue. If/When that's fixed, I'm golden. Thanks so much! On Fri, Jan 10, 2014 at 11:51 AM, Gordon Sim g...@redhat.com wrote: On 01/10/2014 02:26 PM, Shearer, Davin wrote: I took some python code from Apache Dispatch Router and modified it to work with the qpid broker directly without using the dispatch router. [...] This is just using proton, which is great, but the problem is that the dynamic queues have a permanent lifetime policy as demonstrated here: Ok, this now is something that could be fixed in the broker. It could (and arguably should) default to delete-on-close for dynamic nodes. I've raised an issue to track that and will fix it before too long I hope: https://issues.apache.org/jira/browse/QPID-5463 Your initial email suggested you were using the Qpid Java broker? That may default as expected (Rob?). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
Sent from my iPhone On Jan 10, 2014, at 13:49, Ted Ross tr...@redhat.com wrote: On 01/10/2014 01:34 PM, Shearer, Davin wrote: I actually _prefer_ the C++ broker. I was using the java one for the web management. So to get this straight... proton sends '#' to the qpid broker, which in turn generates a node name which it then sends back to the client via the address attribute. Not quite... The '#' is not part of the protocol between the client and broker. What actually happens is that proton (or qpid.messaging) sees the '#' in the address and instead of sending a name across to the broker, sets the dynamic flag in the attach frame, leaving the name blank. The broker sees the dynamic flag, creates a temporary queue, and sends the name of the queue back in the resulting attach frame. Right, but I'm pretty sure the magic '#' thing is a qpid::messaging thing, not a proton thing and as such proton is actually sending '#' on the wire and qpid messaging is doing the magic of creating the dynamic queue. The bit that's missing is to set the lifetime policy to 'delete on close' which Gordon just added into jira. The only thing at is missing is that the qpid broker does not set auto-del on the newly created dynamic queue. If/When that's fixed, I'm golden. Thanks so much! On Fri, Jan 10, 2014 at 11:51 AM, Gordon Sim g...@redhat.com wrote: On 01/10/2014 02:26 PM, Shearer, Davin wrote: I took some python code from Apache Dispatch Router and modified it to work with the qpid broker directly without using the dispatch router. [...] This is just using proton, which is great, but the problem is that the dynamic queues have a permanent lifetime policy as demonstrated here: Ok, this now is something that could be fixed in the broker. It could (and arguably should) default to delete-on-close for dynamic nodes. I've raised an issue to track that and will fix it before too long I hope: https://issues.apache.org/jira/browse/QPID-5463 Your initial email suggested you were using the Qpid Java broker? That may default as expected (Rob?). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
On 01/10/2014 02:03 PM, Davin Shearer wrote: Sent from my iPhone On Jan 10, 2014, at 13:49, Ted Ross tr...@redhat.com wrote: On 01/10/2014 01:34 PM, Shearer, Davin wrote: I actually _prefer_ the C++ broker. I was using the java one for the web management. So to get this straight... proton sends '#' to the qpid broker, which in turn generates a node name which it then sends back to the client via the address attribute. Not quite... The '#' is not part of the protocol between the client and broker. What actually happens is that proton (or qpid.messaging) sees the '#' in the address and instead of sending a name across to the broker, sets the dynamic flag in the attach frame, leaving the name blank. The broker sees the dynamic flag, creates a temporary queue, and sends the name of the queue back in the resulting attach frame. Right, but I'm pretty sure the magic '#' thing is a qpid::messaging thing, not a proton thing and as such proton is actually sending '#' on the wire and qpid messaging is doing the magic of creating the dynamic queue. The bit that's missing is to set the lifetime policy to 'delete on close' which Gordon just added into jira. No. The '#' thing is done the same in proton-messenger as in qpid::messaging. Both use the dynamic flag. Sending '#' on the wire is not standard and the broker would not respond correctly to that. It's important to note that this feature was recently introduced in proton-messenger (as of 0.6). The only thing at is missing is that the qpid broker does not set auto-del on the newly created dynamic queue. If/When that's fixed, I'm golden. Thanks so much! On Fri, Jan 10, 2014 at 11:51 AM, Gordon Sim g...@redhat.com wrote: On 01/10/2014 02:26 PM, Shearer, Davin wrote: I took some python code from Apache Dispatch Router and modified it to work with the qpid broker directly without using the dispatch router. [...] This is just using proton, which is great, but the problem is that the dynamic queues have a permanent lifetime policy as demonstrated here: Ok, this now is something that could be fixed in the broker. It could (and arguably should) default to delete-on-close for dynamic nodes. I've raised an issue to track that and will fix it before too long I hope: https://issues.apache.org/jira/browse/QPID-5463 Your initial email suggested you were using the Qpid Java broker? That may default as expected (Rob?). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
Thanks Ted, I see the new code in proton-c/src/messenger/messenger.c where a node named # gets set to dynamic. Good stuff. On Fri, Jan 10, 2014 at 2:19 PM, Ted Ross tr...@redhat.com wrote: On 01/10/2014 02:03 PM, Davin Shearer wrote: Sent from my iPhone On Jan 10, 2014, at 13:49, Ted Ross tr...@redhat.com wrote: On 01/10/2014 01:34 PM, Shearer, Davin wrote: I actually _prefer_ the C++ broker. I was using the java one for the web management. So to get this straight... proton sends '#' to the qpid broker, which in turn generates a node name which it then sends back to the client via the address attribute. Not quite... The '#' is not part of the protocol between the client and broker. What actually happens is that proton (or qpid.messaging) sees the '#' in the address and instead of sending a name across to the broker, sets the dynamic flag in the attach frame, leaving the name blank. The broker sees the dynamic flag, creates a temporary queue, and sends the name of the queue back in the resulting attach frame. Right, but I'm pretty sure the magic '#' thing is a qpid::messaging thing, not a proton thing and as such proton is actually sending '#' on the wire and qpid messaging is doing the magic of creating the dynamic queue. The bit that's missing is to set the lifetime policy to 'delete on close' which Gordon just added into jira. No. The '#' thing is done the same in proton-messenger as in qpid::messaging. Both use the dynamic flag. Sending '#' on the wire is not standard and the broker would not respond correctly to that. It's important to note that this feature was recently introduced in proton-messenger (as of 0.6). The only thing at is missing is that the qpid broker does not set auto-del on the newly created dynamic queue. If/When that's fixed, I'm golden. Thanks so much! On Fri, Jan 10, 2014 at 11:51 AM, Gordon Sim g...@redhat.com wrote: On 01/10/2014 02:26 PM, Shearer, Davin wrote: I took some python code from Apache Dispatch Router and modified it to work with the qpid broker directly without using the dispatch router. [...] This is just using proton, which is great, but the problem is that the dynamic queues have a permanent lifetime policy as demonstrated here: Ok, this now is something that could be fixed in the broker. It could (and arguably should) default to delete-on-close for dynamic nodes. I've raised an issue to track that and will fix it before too long I hope: https://issues.apache.org/jira/browse/QPID-5463 Your initial email suggested you were using the Qpid Java broker? That may default as expected (Rob?). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org -- Davin Shearer Engineer 8830 Stanford Blvd, Suite 306 Columbia, MD 21045 443-741-4517
Re: How to use lifetime policies
On 10/01/14 18:34, Shearer, Davin wrote: I actually _prefer_ the C++ broker. I was using the java one for the web management. Hi Davin, have you taken a look at the stuff in qpid/tools/src/java? There's a QMF based Web UI for the C++ broker (it'll work with the Java Broker too if you build with ant all which creates the QMF plugin - it's not quite as comprehensive as the native UI for the Java Broker but provides consistency between the C++ and Java Brokers). The backend is a QMF based REST API, you fire it up going to the bin directory and doing ./QpidRestAPI Point a browser at it (default uname/pword is the traditional admin/admin - change it in qpid/tools/src/java/bin/qpid-web/authentication/account.properties) You can actually navigate to different broker instances through the UI by specifying different connections. It should run on everything from IE6 to an iPhone. If you want web management for the C++ broker you might find it useful. Caveat - it's currently limited to basic authentication as I've not had the time/energy/motivation to do anything fancier. Regards, Frase - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
On 01/10/2014 06:34 PM, Shearer, Davin wrote: The only thing at is missing is that the qpid broker does not set auto-del on the newly created dynamic queue. If/When that's fixed, I'm golden. Done[1], as penance for my earlier mistake! [1] https://svn.apache.org/r1557262 - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
Wow, I didn't know about this but I'll definitely try it out. Sent from my iPhone On Jan 10, 2014, at 15:46, Fraser Adams fraser.ad...@blueyonder.co.uk wrote: On 10/01/14 18:34, Shearer, Davin wrote: I actually _prefer_ the C++ broker. I was using the java one for the web management. Hi Davin, have you taken a look at the stuff in qpid/tools/src/java? There's a QMF based Web UI for the C++ broker (it'll work with the Java Broker too if you build with ant all which creates the QMF plugin - it's not quite as comprehensive as the native UI for the Java Broker but provides consistency between the C++ and Java Brokers). The backend is a QMF based REST API, you fire it up going to the bin directory and doing ./QpidRestAPI Point a browser at it (default uname/pword is the traditional admin/admin - change it in qpid/tools/src/java/bin/qpid-web/authentication/account.properties) You can actually navigate to different broker instances through the UI by specifying different connections. It should run on everything from IE6 to an iPhone. If you want web management for the C++ broker you might find it useful. Caveat - it's currently limited to basic authentication as I've not had the time/energy/motivation to do anything fancier. Regards, Frase - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
Thanks for the help and the fix! Sent from my iPhone On Jan 10, 2014, at 16:20, Gordon Sim g...@redhat.com wrote: On 01/10/2014 06:34 PM, Shearer, Davin wrote: The only thing at is missing is that the qpid broker does not set auto-del on the newly created dynamic queue. If/When that's fixed, I'm golden. Done[1], as penance for my earlier mistake! [1] https://svn.apache.org/r1557262 - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: How to use lifetime policies
Hi Davin,
On 9 January 2014 19:46, Shearer, Davin dshea...@novetta.com wrote:
Hello,
Is there a way, using proton and QMF to create a queue that has a lifetime
policy other than permanent? If so, how?
I'm not aware if/how you can set the lifetime policies using Proton,
hopefully someone who is can chime in. To assist them, could you indicate
which part of Proton-c you are using specifically, e.g. Messenger or the
underlying Engine.
The Java broker doesn't support QMF by default though Fraser Adams did
write a plugin for it to add a level of support for it via the AMQP 0-10
clients and tools to go along with the GUI he put together for the C++
broker originally, but I'm not sure if that helps here as I have little
experience with it.
I am trying to build a transport library in C using proton with support for
a qpid broker. What I need to do is create a reply queue. The AMQP 1.0
spec covers creating Dynamic Reply-To. The problem I'm having is in
garbage collection of the abandoned queues as these queues have a permanent
lifetime policy. If I were using qpid (rather than proton) I could simply
have the client create the queue using a name of its choice with the
DeleteOnClose lifetime policy and I also set exclusive to true and I get
exactly what I need, but it this ties me to qpid.
Instead, what I want to do is use proton for the AMQP 1.0 messaging and for
queue management use the vendor-specific management framework (so for qpid,
it is QMF). This means that if I need to support other vendor's broker, I
extend the queue management functions with support for that vendor's
specific management framework and the actual messaging (since its AMQP 1.0)
just uses proton.
My thought is to use the qpid java broker's REST interface for queue
creation. It seems that I can create queues with the DeleteOn* lifetime
policies using REST, but then the queues are immediately deleted.
Just to clear up some possible confusion at this point, the Lifetime Policy
reported via the Java broker REST interface doesn't reflect the AMQP 1.0
node LifetimePolicies (not yet at least). The value is currently able to be
either PERMANENT (the default) or AUTO_DELETE (hailing from AMQP 0-x).
From a squint at the code, I think it probably displays the incorrect value
for a dynamically created queue created on a 1.0 connection as a result,
always indicating it is permanent when it may not be (because it doesnt get
marked as autodelete, and is instead removed via a different mechanism
based on the lifetime policy).
Also, from the same quick squint at the code, the broker only appears to
currently supports the Delete On Close policy (which is also what it uses
if none is specified).
bash$ for policy in DeleteOnClose DeleteOnNoLinks DeleteOnNoLinksOfMessages
DeleteOMessages; do echo Policy: $policy; curl -X PUT -u
webadmin:webadmin -d {\durable\:true,\lifetimePolicy\:\${policy}\}
http://localhost:8080/rest/queue/default/test_queue; curl -X GET -u
webadmin:webadmin http://localhost:8080/rest/queue/default/test_queue;
echo; done
Policy: DeleteOnClose
[ ]
Policy: DeleteOnNoLinks
[ ]
Policy: DeleteOnNoLinksOfMessages
[ ]
Policy: DeleteOMessages
[ ]
Continuing from the previous comments, the REST interface doenst support
specifying lifetimePolicy with values like that, not yet at least. It
appears you can specify AUTO_DELETE when creating the queue through REST
although I must admit I have never tried that, and definitely don't know
how it would react with a 1.0 consumer since the effect it would have is
different than that performed by the lifetime policy normally.
Also if I just create the queue, then try to change attributes later, the
changes are ignored:
curl PUT -u webadmin:webadmin -d '{}'
http://localhost:8080/rest/queue/default/test_queue
curl -X GET -u webadmin:webadmin
http://localhost:8080/rest/queue/default/test_queue
[ {
id : f280eb22-8110-3d2b-91d3-f68192c8d1b7,
name : test_queue,
state : ACTIVE,
durable : false,
lifetimePolicy : PERMANENT,
type : standard,
exclusive : false,
maximumDeliveryAttempts : 0,
queueFlowControlSizeBytes : 0,
queueFlowResumeSizeBytes : 0,
queueFlowStopped : false,
alertThresholdMessageAge : 0,
alertThresholdMessageSize : 0,
alertThresholdQueueDepthBytes : 0,
alertThresholdQueueDepthMessages : 0,
alertRepeatGap : 3,
statistics : {
unacknowledgedBytes : 0,
bindingCount : 0,
queueDepthBytes : 0,
queueDepthMessages : 0,
totalEnqueuedBytes : 0,
persistentDequeuedBytes : 0,
consumerCount : 0,
unacknowledgedMessages : 0,
consumerCountWithCredit : 0,
totalDequeuedBytes : 0,
totalDequeuedMessages : 0,
totalEnqueuedMessages : 0,
persistentEnqueuedMessages : 0,
persistentEnqueuedBytes : 0,
persistentDequeuedMessages : 0
}
} ]
curl -X PUT -u webadmin:webadmin -d '{durable:true}'
