Re: Qpid Dispatch Router component

[Gordon Sim]

It was pointed out to me that sloppy wording on my part may have led to 
an impression other than the one intended. At the risk of digging a 
bigger hole, I'd like to try to correct that.


On 10/11/2013 12:36 PM, Gordon Sim wrote:

As I said before, my interest is not in reforming OASIS. My interest is
in the Qpid project and in demonstrable interoperability and choice.


The statement above was simply intended to clarify the purpose of my 
comments in this thread.


I was trying to ensure the conversation did not veer off in an 
unintended direction rather than voicing any more general opinion.


I have an interest in *anything* that can deliver practical 
interoperability and choice for messaging users.


I apologise for the lack of precision in my original email and any 
misunderstanding that may have caused.


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Gordon Sim]

On 10/12/2013 11:40 AM, Fraser Adams wrote:

a primary concern and motivation has to be standardisation,
particularly Open Standards and they must be seen to be platform
neutral - so fundamentally AMQP first and Qpid second. To me that's a
reasonable position too because, as has been expressed elsewhere, one
of the compelling reasons of "why AMQP" is because it's an Open
Standard and *should be* interoperable.


I don't disagree with any of that. Compliance with the current 
specification and a commitment to demonstrable interoperability around 
that is, and must be, fundamental to Qpid.


I do also very much want to see interoperability extend to areas not 
directly covered by this specification.


What I am uncomfortable with is the view that any initiative, wherever 
and however it was begun, has from the start some unassailable dominion 
over the area it stakes out for itself.



Sadly that wasn't really achieved for 0.10, but it certainly seems to
be the case for 1.0 which I feel we ought to embrace with a passion.


The history of the 0-10 specification is actually interesting to 
consider here. Though it was the de-jure standard, having passed by 
unanimous vote through the old working group process, it was clear from 
the start that there was no real consensus behind it (even from some of 
those voting, not to mention anyone not represented in the process). 
Though supported by all Qpid components, it was implemented by nothing 
else. That is a pattern I don't want to see repeated.


While achieving real consensus is hard, slow and often frustrating, 
'official' standards that aren't widely supported are of limited value.


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Fraser Adams]

I'd like to wade in slightly here if I may as I've been following this 
thread with great interest.


I guess that I might be in a vaguely interesting position as although my 
employers are interested in and make use of AMQP/Qpid my contributions 
to this group have been entirely at a personal level and not on behalf 
of an employer, simply because I think it's interesting and worthwhile.


The reason I mention that is in relation to organisations like OASIS.

So I find myself rather torn, I really agree with Gordon and Ted on one 
hand about innovation and driving things bottom up (decent standards 
tend to come with "reference implementations") but on the other hand 
I've got a lot of sympathy with Rob and his colleagues who very much 
have a position that a primary concern and motivation has to be 
standardisation, particularly Open Standards and they must be seen to be 
platform neutral - so fundamentally AMQP first and Qpid second. To me 
that's a reasonable position too because, as has been expressed 
elsewhere, one of the compelling reasons of "why AMQP" is because it's 
an Open Standard and *should be* interoperable. Sadly that wasn't really 
achieved for 0.10, but it certainly seems to be the case for 1.0 which I 
feel we ought to embrace with a passion.


But why I mention OASIS, well TBH I have a bit of a problem with them in 
the sense that they are putting together Open Standards for sure, but I 
don't really believe that the process for achieving said standards seems 
to be in any way open and transparent!!


One of the reasons that I like Apache is the fundamental doctrine of 
openness and having a licence that is non-restrictive, so products can 
be used most anywhere. We're in a position where organisations and 
individuals can contribute and is essentially democratic/meritocratic - 
that feels like a healthy place to be in the 21st Century.


The same can't be said of OASIS, so someone like myself just might have 
vaguely relevant experiences with management say, but there's no real 
avenue for contribution because it's a bit of a "closed shop" made up of 
fairly large organisations who have to pay a non-trivial fee for 
membership (I'm keen, but I'm not *that* keen :-)). Now I can sympathise 
a bit and I'm sure that part of the motivation is paranoia about 
avoidance of potential submarine patents probably driven by some of the 
big players, but I can't help believe that there might be better and 
more open ways to achieve the same goals (those guys can afford some 
decent lawyers after all :-)).


One of the things about OASIS that worries me as an outsider is how much 
of the decision making ends up getting driven by some corporate agenda - 
after all if you've got a product to sell there might be benefits in 
shaping the direction of a standard to best suit your architecture or 
roadmap and potentially give some commercial advantage. I'm not accusing 
any organisation of doing such a thing and I'm sure that it's all lovely 
and altruistic, but without openness and transparency it's hard to say 
for sure. If I'm honest given where AMQP sits in the protocol stack I 
often wonder why the hell it's OASIS and not say IETF that holds the 
standard, from what I can see OASIS seems to sit rather more at the 
"higher end of the stack" formalising B2B interoperability whereas AMQP 
1.0 seems to have evolved into something arguably closer to a network 
layer protocol (after all we're on a thread about routers :-)).


Anyway this probably hasn't added much to the discussion and has turned 
a bit into a polemic about openness and transparency so sorry about 
that, but I'm quite passionate that it's healthier for an Open Source 
model to encompass more than just "a bit of code" and I think that when 
we're talking about Open Standards the Open Source model should extend 
out to ideas and architectures as well - you've probably noted from my 
"how it all hangs together" post that I'm keen for a visible, open, 
modular architecture :-)


Bottom line is what  I'd *really* like to see is Qpid components working 
towards and enabling Open Standards and I'd like to see OASIS opening up 
a bit and being more practical. FWIW I'm a little nervous about the 
Management Specification and probably won't feel happy until we've got 
something kickable, I've read it a couple of times and it still feels a 
little abstract and open to interpretation, but that's largely in the 
nature of these things, which is why reference implementations are good 
and should help the standards evolve.


Regards,
Frase


On 11/10/13 21:36, Gordon Sim wrote:

On 10/11/2013 06:33 PM, William Henry wrote:
Yes Ted, and Gordon I believe, has been involved in trying to get 
this feedback based on our experiences.


Simplify to clarify: I am no longer involved in any way with OASIS.

Further, any opinions I expressed while I was there were my own and 
did not in any way represent the views of my employer or any of my 
colleagues.


--

Re: Qpid Dispatch Router component

[Fraser Adams]

On the topic of names and perhaps following the same track as particles 
with Proton, can I suggest "Boson" in honour of the forthcoming Nobel 
award to Professor Peter Higgs. It kind of seems appropriate on a couple 
of levels?


Frase

On 11/10/13 13:47, Gordon Sim wrote:

On 10/11/2013 01:32 PM, Ted Ross wrote:

I have no love for the name.  But it has to be called *something* :)


Indeed, and I reiterate how hard good naming is.

Assuming 'Qpid Router' is undesirable as either making to big a claim 
(there will never be any other Qpid thing that routes) or provides 
insufficient 'identity', one option might be to follow the elementary 
particles approach started with proton. How about 'gluon' or perhaps 
'graviton'? :-)


I personally think a simple DX or QDX as an opaque name would be 
better as it more obviously acknowledges that its merely an identifier 
with little or no underlying description.


Again, this should in no way imply any opposition to proceeding as is. 
It is just an observation/suggestion.


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Gordon Sim]

On 10/11/2013 06:33 PM, William Henry wrote:

Yes Ted, and Gordon I believe, has been involved in trying to get this feedback 
based on our experiences.


Simplify to clarify: I am no longer involved in any way with OASIS.

Further, any opinions I expressed while I was there were my own and did 
not in any way represent the views of my employer or any of my colleagues.


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey]

On 11 October 2013 19:43, Ted Ross  wrote:

> Steve,
>
> I'm a member of the OASIS technical committee that is working on both the
> addressing and management specifications.  I see my role there as
> tester/implementer.  Both specifications have a long way to go before they
> are usable IMO.  It is my plan that Dispatch will become compliant with
> both specifications, but it is a two-way street.  The TC needs also to
> respect the needs of implementers.
>
>
I absolutely agree... that is the point of the TCs.  It would be good if
all of us who have questions, comments, etc raise them to the TC, and those
of us who are members ensure that these concerns/comments/questions are
acted upon and also fedback to this list.

None of the work in progress is solidified yet precisely because it needs
feedback from implementers and users.  The work won't be finalised until we
have interoperable implementations from different sources.

If you've raised concerns around the two pieces of work in question and you
do not think they have been adequately addressed, you should definitely
send a chaser e-mail.  If there is stuff you haven't raised yet, then
please do so.

To be clear, the latest working draft of the addressing specification is
here:

https://www.oasis-open.org/committees/document.php?document_id=50701&wg_abbrev=amqp

And the latest draft of the management specification is here:

https://www.oasis-open.org/committees/document.php?document_id=50101&wg_abbrev=amqp

Those who are not members of the TCs can raise comments/questions through
the public comment list (see:
https://www.oasis-open.org/committees/comments/index.php?wg_abbrev=amqp)

Hope this helps,
Rob

-Ted
>
>
>
> On 10/11/2013 01:18 PM, Steve Huston wrote:
>
>> Hi William,
>>
>> Thanks for expressing your view. And no reasonable person would discount
>> it.
>>
>> Speaking for my own points, I would not argue that Dispatch (or any
>> trailblazing development) should _wait_ for OASIS. However, since there are
>> 2 areas of active work in OASIS that are related, and Dispatch is quickly
>> gaining experience and backing, I would very much like to see Dispatch
>> development folks actively involved in informing the OASIS technical
>> committees of their experience and help to get the eventual standard(s)
>> worked out in a way that's been proven in the field.
>>
>> -Steve
>>
>>  -Original Message-
>>> From: William Henry [mailto:whe...@redhat.com]
>>> Sent: Friday, October 11, 2013 12:49 PM
>>> To: users@qpid.apache.org
>>> Subject: Re: Qpid Dispatch Router component
>>>
>>>
>>> I'd also add, because I'm sure that some of you wonder why is William
>>> weighing in? What has he to do with AMQP/Qpid anyway besides being
>>> historically linked to it?
>>>
>>> I've travelled globally talking about AMQP and Qpid to investment banks,
>>> government agencies, telecoms, animation studios, stock exchanges etc.
>>> etc.
>>> (strangers on planes ;-)  I'm sure others have done more to help promote
>>> AMQP and Qpid but, despite my seemingly lack of, or invisible, community
>>> involvement of late, I bet that list of others-that-have-done-more-to-
>>> promote it is actually not a very large list. I can think of a few. I
>>> have other
>>> technology areas I focus on but I continue to be involved in AMQP/Qpid
>>> because I get pulled back in, and I still "believe". There is a massive
>>> opportunity for AMQP that is more than just simple MOM.
>>>
>>> So my 2 cents at a high level:
>>> 1) People love the AMQP story
>>> 2) People love the AMQP network story - with a variety of AMQP based
>>> assets.
>>> 3) As we've introduced the concept of what Dispatch does, people love it!
>>> Did I say love it?! And they do not see it as tangential to AMQP. But
>>> massively
>>> complimentary. A Dispatch type network with (various) brokers backing it
>>> up
>>> nearer to applications - that have different client interfaces (JMS, JCA,
>>> Python, C/C++, Ruby etc.)
>>> 4) The cloud will and is already playing a big part in AMQP adoption. We
>>> need
>>> to capitalize.
>>> 5) People want solutions now! Management now. Addressing nailed down
>>> now.
>>>
>>> Best,
>>> William
>>>
>>> - Original Message -
>>>
>>>> +1. Great post Gordon. Again something to be captured in Qpid
>>>> +community
>>>> pages/documentation.
>>>>
>>&

Re: Qpid Dispatch Router component

[Ted Ross]

Steve,

I'm a member of the OASIS technical committee that is working on both 
the addressing and management specifications.  I see my role there as 
tester/implementer.  Both specifications have a long way to go before 
they are usable IMO.  It is my plan that Dispatch will become compliant 
with both specifications, but it is a two-way street.  The TC needs also 
to respect the needs of implementers.


-Ted


On 10/11/2013 01:18 PM, Steve Huston wrote:

Hi William,

Thanks for expressing your view. And no reasonable person would discount it.

Speaking for my own points, I would not argue that Dispatch (or any 
trailblazing development) should _wait_ for OASIS. However, since there are 2 
areas of active work in OASIS that are related, and Dispatch is quickly gaining 
experience and backing, I would very much like to see Dispatch development 
folks actively involved in informing the OASIS technical committees of their 
experience and help to get the eventual standard(s) worked out in a way that's 
been proven in the field.

-Steve


-Original Message-
From: William Henry [mailto:whe...@redhat.com]
Sent: Friday, October 11, 2013 12:49 PM
To: users@qpid.apache.org
Subject: Re: Qpid Dispatch Router component


I'd also add, because I'm sure that some of you wonder why is William
weighing in? What has he to do with AMQP/Qpid anyway besides being
historically linked to it?

I've travelled globally talking about AMQP and Qpid to investment banks,
government agencies, telecoms, animation studios, stock exchanges etc. etc.
(strangers on planes ;-)  I'm sure others have done more to help promote
AMQP and Qpid but, despite my seemingly lack of, or invisible, community
involvement of late, I bet that list of others-that-have-done-more-to-
promote it is actually not a very large list. I can think of a few. I have other
technology areas I focus on but I continue to be involved in AMQP/Qpid
because I get pulled back in, and I still "believe". There is a massive
opportunity for AMQP that is more than just simple MOM.

So my 2 cents at a high level:
1) People love the AMQP story
2) People love the AMQP network story - with a variety of AMQP based
assets.
3) As we've introduced the concept of what Dispatch does, people love it!
Did I say love it?! And they do not see it as tangential to AMQP. But massively
complimentary. A Dispatch type network with (various) brokers backing it up
nearer to applications - that have different client interfaces (JMS, JCA,
Python, C/C++, Ruby etc.)
4) The cloud will and is already playing a big part in AMQP adoption. We need
to capitalize.
5) People want solutions now! Management now. Addressing nailed down
now.

Best,
William

- Original Message -

+1. Great post Gordon. Again something to be captured in Qpid
+community
pages/documentation.

I'd also add we need to be innovating fast around AMQP now that we have

1.0.

We can't always wait on the OASIS process. It has taken a very long
time to get here (not OASIS). There were some significant changes along

the way.

Users are looking for solutions today.

I understand there is a risk of getting ahead of ourselves in certain
areas and having to backtrack. However people that have been investing
in AMQP are looking for solutions to their problems ASAP and can't
wait until every area has been worked out.

I agree there is a risk of a high cost to "fixing" addressing and management.
How can we get there faster? Surely innovations/projects like Dispatch
can only help in driving the standardization faster as it and other
artifacts hit issues.  I think Dispatch has already helped drive the
addressing discussion.

Let's press forward.

William

- Original Message -

I don't claim to speak with any authority on Dispatch Router, but
fwiw, here's my answer to these two specific questions:

On 10/10/2013 04:20 PM, Rob Godfrey wrote:

How does a router differ from a broker?

I tend to have a more general view of what 'broker' might mean, but
in the context of statements that Dispatch Router 'is not a broker',
what is meant is the type of intermediary as exemplified by the two
existing Qpid brokers and others like them.

As I see it, there are two key differences between Dispatch Router
and these.

The first is that a principle of the Dispatch Router is that it does
not 'take ownership' of messages. This is in contrast to most
brokers where the publisher transfers responsibility for messages to
the broker and the broker then undertakes to deliver them allowing
the producer to forget about them. With Dispatch Router, the idea is
that it will route those messages, but any guarantee regarding
acceptance of those messages comes from the consumer(s) of the

message, not from the intermediary.

The second distinction is that Dispatch Router is from the start
designed to be a small piece in a distributed routing network 

Re: Qpid Dispatch Router component

[William Henry]

Thanks Steve,

Yes Ted, and Gordon I believe, has been involved in trying to get this feedback 
based on our experiences. 

As you know I was involved in the TC for a while but I got dragged elsewhere. 
Also, perhaps I'm not as patient as I used to be for standards body work. It's 
a tough job and requires a lot of patience. Maybe too much ;-)

William

- Original Message -
> Hi William,
> 
> Thanks for expressing your view. And no reasonable person would discount it.
> 
> Speaking for my own points, I would not argue that Dispatch (or any
> trailblazing development) should _wait_ for OASIS. However, since there are
> 2 areas of active work in OASIS that are related, and Dispatch is quickly
> gaining experience and backing, I would very much like to see Dispatch
> development folks actively involved in informing the OASIS technical
> committees of their experience and help to get the eventual standard(s)
> worked out in a way that's been proven in the field.
> 
> -Steve
> 
> > -Original Message-
> > From: William Henry [mailto:whe...@redhat.com]
> > Sent: Friday, October 11, 2013 12:49 PM
> > To: users@qpid.apache.org
> > Subject: Re: Qpid Dispatch Router component
> > 
> > 
> > I'd also add, because I'm sure that some of you wonder why is William
> > weighing in? What has he to do with AMQP/Qpid anyway besides being
> > historically linked to it?
> > 
> > I've travelled globally talking about AMQP and Qpid to investment banks,
> > government agencies, telecoms, animation studios, stock exchanges etc. etc.
> > (strangers on planes ;-)  I'm sure others have done more to help promote
> > AMQP and Qpid but, despite my seemingly lack of, or invisible, community
> > involvement of late, I bet that list of others-that-have-done-more-to-
> > promote it is actually not a very large list. I can think of a few. I have
> > other
> > technology areas I focus on but I continue to be involved in AMQP/Qpid
> > because I get pulled back in, and I still "believe". There is a massive
> > opportunity for AMQP that is more than just simple MOM.
> > 
> > So my 2 cents at a high level:
> > 1) People love the AMQP story
> > 2) People love the AMQP network story - with a variety of AMQP based
> > assets.
> > 3) As we've introduced the concept of what Dispatch does, people love it!
> > Did I say love it?! And they do not see it as tangential to AMQP. But
> > massively
> > complimentary. A Dispatch type network with (various) brokers backing it up
> > nearer to applications - that have different client interfaces (JMS, JCA,
> > Python, C/C++, Ruby etc.)
> > 4) The cloud will and is already playing a big part in AMQP adoption. We
> > need
> > to capitalize.
> > 5) People want solutions now! Management now. Addressing nailed down
> > now.
> > 
> > Best,
> > William
> > 
> > - Original Message -
> > > +1. Great post Gordon. Again something to be captured in Qpid
> > > +community
> > > pages/documentation.
> > >
> > > I'd also add we need to be innovating fast around AMQP now that we have
> > 1.0.
> > > We can't always wait on the OASIS process. It has taken a very long
> > > time to get here (not OASIS). There were some significant changes along
> > the way.
> > > Users are looking for solutions today.
> > >
> > > I understand there is a risk of getting ahead of ourselves in certain
> > > areas and having to backtrack. However people that have been investing
> > > in AMQP are looking for solutions to their problems ASAP and can't
> > > wait until every area has been worked out.
> > >
> > > I agree there is a risk of a high cost to "fixing" addressing and
> > > management.
> > > How can we get there faster? Surely innovations/projects like Dispatch
> > > can only help in driving the standardization faster as it and other
> > > artifacts hit issues.  I think Dispatch has already helped drive the
> > > addressing discussion.
> > >
> > > Let's press forward.
> > >
> > > William
> > >
> > > - Original Message -
> > > > I don't claim to speak with any authority on Dispatch Router, but
> > > > fwiw, here's my answer to these two specific questions:
> > > >
> > > > On 10/10/2013 04:20 PM, Rob Godfrey wrote:
> > > > > How does a router differ from a broker?
> > > >
> > > > I tend to have a more general vie

RE: Qpid Dispatch Router component

[Steve Huston]

Hi William,

Thanks for expressing your view. And no reasonable person would discount it.

Speaking for my own points, I would not argue that Dispatch (or any 
trailblazing development) should _wait_ for OASIS. However, since there are 2 
areas of active work in OASIS that are related, and Dispatch is quickly gaining 
experience and backing, I would very much like to see Dispatch development 
folks actively involved in informing the OASIS technical committees of their 
experience and help to get the eventual standard(s) worked out in a way that's 
been proven in the field.

-Steve

> -Original Message-
> From: William Henry [mailto:whe...@redhat.com]
> Sent: Friday, October 11, 2013 12:49 PM
> To: users@qpid.apache.org
> Subject: Re: Qpid Dispatch Router component
> 
> 
> I'd also add, because I'm sure that some of you wonder why is William
> weighing in? What has he to do with AMQP/Qpid anyway besides being
> historically linked to it?
> 
> I've travelled globally talking about AMQP and Qpid to investment banks,
> government agencies, telecoms, animation studios, stock exchanges etc. etc.
> (strangers on planes ;-)  I'm sure others have done more to help promote
> AMQP and Qpid but, despite my seemingly lack of, or invisible, community
> involvement of late, I bet that list of others-that-have-done-more-to-
> promote it is actually not a very large list. I can think of a few. I have 
> other
> technology areas I focus on but I continue to be involved in AMQP/Qpid
> because I get pulled back in, and I still "believe". There is a massive
> opportunity for AMQP that is more than just simple MOM.
> 
> So my 2 cents at a high level:
> 1) People love the AMQP story
> 2) People love the AMQP network story - with a variety of AMQP based
> assets.
> 3) As we've introduced the concept of what Dispatch does, people love it!
> Did I say love it?! And they do not see it as tangential to AMQP. But 
> massively
> complimentary. A Dispatch type network with (various) brokers backing it up
> nearer to applications - that have different client interfaces (JMS, JCA,
> Python, C/C++, Ruby etc.)
> 4) The cloud will and is already playing a big part in AMQP adoption. We need
> to capitalize.
> 5) People want solutions now! Management now. Addressing nailed down
> now.
> 
> Best,
> William
> 
> - Original Message -
> > +1. Great post Gordon. Again something to be captured in Qpid
> > +community
> > pages/documentation.
> >
> > I'd also add we need to be innovating fast around AMQP now that we have
> 1.0.
> > We can't always wait on the OASIS process. It has taken a very long
> > time to get here (not OASIS). There were some significant changes along
> the way.
> > Users are looking for solutions today.
> >
> > I understand there is a risk of getting ahead of ourselves in certain
> > areas and having to backtrack. However people that have been investing
> > in AMQP are looking for solutions to their problems ASAP and can't
> > wait until every area has been worked out.
> >
> > I agree there is a risk of a high cost to "fixing" addressing and 
> > management.
> > How can we get there faster? Surely innovations/projects like Dispatch
> > can only help in driving the standardization faster as it and other
> > artifacts hit issues.  I think Dispatch has already helped drive the
> > addressing discussion.
> >
> > Let's press forward.
> >
> > William
> >
> > - Original Message -
> > > I don't claim to speak with any authority on Dispatch Router, but
> > > fwiw, here's my answer to these two specific questions:
> > >
> > > On 10/10/2013 04:20 PM, Rob Godfrey wrote:
> > > > How does a router differ from a broker?
> > >
> > > I tend to have a more general view of what 'broker' might mean, but
> > > in the context of statements that Dispatch Router 'is not a broker',
> > > what is meant is the type of intermediary as exemplified by the two
> > > existing Qpid brokers and others like them.
> > >
> > > As I see it, there are two key differences between Dispatch Router
> > > and these.
> > >
> > > The first is that a principle of the Dispatch Router is that it does
> > > not 'take ownership' of messages. This is in contrast to most
> > > brokers where the publisher transfers responsibility for messages to
> > > the broker and the broker then undertakes to deliver them allowing
> > > the producer to forget about them. With Dispatch Router, the idea is
>

Re: Qpid Dispatch Router component

[William Henry]

I'd also add, because I'm sure that some of you wonder why is William weighing 
in? What has he to do with AMQP/Qpid anyway besides being historically linked 
to it?

I've travelled globally talking about AMQP and Qpid to investment banks, 
government agencies, telecoms, animation studios, stock exchanges etc. etc. 
(strangers on planes ;-)  I'm sure others have done more to help promote AMQP 
and Qpid but, despite my seemingly lack of, or invisible, community involvement 
of late, I bet that list of others-that-have-done-more-to-promote it is 
actually not a very large list. I can think of a few. I have other technology 
areas I focus on but I continue to be involved in AMQP/Qpid because I get 
pulled back in, and I still "believe". There is a massive opportunity for AMQP 
that is more than just simple MOM.  

So my 2 cents at a high level:
1) People love the AMQP story
2) People love the AMQP network story - with a variety of AMQP based assets.
3) As we've introduced the concept of what Dispatch does, people love it! Did I 
say love it?! And they do not see it as tangential to AMQP. But massively 
complimentary. A Dispatch type network with (various) brokers backing it up 
nearer to applications - that have different client interfaces (JMS, JCA, 
Python, C/C++, Ruby etc.)  
4) The cloud will and is already playing a big part in AMQP adoption. We need 
to capitalize. 
5) People want solutions now! Management now. Addressing nailed down now.   
 
Best,
William

- Original Message -
> +1. Great post Gordon. Again something to be captured in Qpid community
> pages/documentation.
> 
> I'd also add we need to be innovating fast around AMQP now that we have 1.0.
> We can't always wait on the OASIS process. It has taken a very long time to
> get here (not OASIS). There were some significant changes along the way.
> Users are looking for solutions today.
> 
> I understand there is a risk of getting ahead of ourselves in certain areas
> and having to backtrack. However people that have been investing in AMQP are
> looking for solutions to their problems ASAP and can't wait until every area
> has been worked out.
> 
> I agree there is a risk of a high cost to "fixing" addressing and management.
> How can we get there faster? Surely innovations/projects like Dispatch can
> only help in driving the standardization faster as it and other artifacts
> hit issues.  I think Dispatch has already helped drive the addressing
> discussion.
> 
> Let's press forward.
> 
> William
> 
> - Original Message -
> > I don't claim to speak with any authority on Dispatch Router, but fwiw,
> > here's my answer to these two specific questions:
> > 
> > On 10/10/2013 04:20 PM, Rob Godfrey wrote:
> > > How does a router differ from a broker?
> > 
> > I tend to have a more general view of what 'broker' might mean, but in
> > the context of statements that Dispatch Router 'is not a broker', what
> > is meant is the type of intermediary as exemplified by the two existing
> > Qpid brokers and others like them.
> > 
> > As I see it, there are two key differences between Dispatch Router and
> > these.
> > 
> > The first is that a principle of the Dispatch Router is that it does not
> > 'take ownership' of messages. This is in contrast to most brokers where
> > the publisher transfers responsibility for messages to the broker and
> > the broker then undertakes to deliver them allowing the producer to
> > forget about them. With Dispatch Router, the idea is that it will route
> > those messages, but any guarantee regarding acceptance of those messages
> > comes from the consumer(s) of the message, not from the intermediary.
> > 
> > The second distinction is that Dispatch Router is from the start
> > designed to be a small piece in a distributed routing network of
> > interconnected components.
> > 
> > Now, there is nothing that says a broker can't do these things. In fact
> > I added code a while back to the Qpid c++ broker that does the former -
> > relays messages through it, providing end-to-end guarantees.
> > 
> > Likewise the distributed architecture has its roots in things like the
> > federation in the Qpid c++ broker (and similar solutions by other brokers).
> > 
> > The Router however is setting out from the start to do *only* these
> > things. It is based on the idea that these are separate concerns that
> > can be split apart allowing 'brokers' to focus on queueing, persistence,
> > transactions, augmented functionality such as LVQ patterns etc, and that
> > the 'federation' logic can be provided in a separate component that can
> > hook up different brokers into a federation as well as hooking in other
> > types of AMQP based services and allow clients to connect directly to
> > the router as well.
> > 
> > By focusing on one aspect its thought components can do a better job.
> > E.g. as I understand it, the Dispatch Router aims to address some of the
> > weaknesses of the federation support in qpidd, by providing fo

Re: Qpid Dispatch Router component

[William Henry]

+1. Great post Gordon. Again something to be captured in Qpid community 
pages/documentation.

I'd also add we need to be innovating fast around AMQP now that we have 1.0. We 
can't always wait on the OASIS process. It has taken a very long time to get 
here (not OASIS). There were some significant changes along the way. Users are 
looking for solutions today.

I understand there is a risk of getting ahead of ourselves in certain areas and 
having to backtrack. However people that have been investing in AMQP are 
looking for solutions to their problems ASAP and can't wait until every area 
has been worked out. 

I agree there is a risk of a high cost to "fixing" addressing and management. 
How can we get there faster? Surely innovations/projects like Dispatch can only 
help in driving the standardization faster as it and other artifacts hit 
issues.  I think Dispatch has already helped drive the addressing discussion.

Let's press forward.

William

- Original Message -
> I don't claim to speak with any authority on Dispatch Router, but fwiw,
> here's my answer to these two specific questions:
> 
> On 10/10/2013 04:20 PM, Rob Godfrey wrote:
> > How does a router differ from a broker?
> 
> I tend to have a more general view of what 'broker' might mean, but in
> the context of statements that Dispatch Router 'is not a broker', what
> is meant is the type of intermediary as exemplified by the two existing
> Qpid brokers and others like them.
> 
> As I see it, there are two key differences between Dispatch Router and
> these.
> 
> The first is that a principle of the Dispatch Router is that it does not
> 'take ownership' of messages. This is in contrast to most brokers where
> the publisher transfers responsibility for messages to the broker and
> the broker then undertakes to deliver them allowing the producer to
> forget about them. With Dispatch Router, the idea is that it will route
> those messages, but any guarantee regarding acceptance of those messages
> comes from the consumer(s) of the message, not from the intermediary.
> 
> The second distinction is that Dispatch Router is from the start
> designed to be a small piece in a distributed routing network of
> interconnected components.
> 
> Now, there is nothing that says a broker can't do these things. In fact
> I added code a while back to the Qpid c++ broker that does the former -
> relays messages through it, providing end-to-end guarantees.
> 
> Likewise the distributed architecture has its roots in things like the
> federation in the Qpid c++ broker (and similar solutions by other brokers).
> 
> The Router however is setting out from the start to do *only* these
> things. It is based on the idea that these are separate concerns that
> can be split apart allowing 'brokers' to focus on queueing, persistence,
> transactions, augmented functionality such as LVQ patterns etc, and that
> the 'federation' logic can be provided in a separate component that can
> hook up different brokers into a federation as well as hooking in other
> types of AMQP based services and allow clients to connect directly to
> the router as well.
> 
> By focusing on one aspect its thought components can do a better job.
> E.g. as I understand it, the Dispatch Router aims to address some of the
> weaknesses of the federation support in qpidd, by providing for
> redundant, fault-tolerant routes without duplication of messages,
> adapting to failures and dynamically computing the best path between two
> endpoints.
> 
> I confess, for all its faults, federation has been one of my favourite
> features of the c++ broker because of the potential I see in it. That is
> why I started off adding some 1.0 related improvements. However I think
> Ted's thinking on Dispatch Router is quite compelling (though I also
> feel a lot of the necessary detail is still missing and it still needs
> to be proven).
> 
> For my part, I am excited to see how this develops, am keen to take part
> if I can and would encourage anyone with an interest to chip in with
> ideas, feedback, criticism, questions, code, use cases or whatever else.
> This began as Ted's 'brainchild' - and most things begin as someone's
> idea - but it really is now open to anyone and everyone who is
> interested. Get involved and shape the direction!
> 
> > Would you expect any special client APIs to form part of the router
> > package or not?
> 
> I'm not entirely sure if I understand the question, but there are again
> two points I would make.
> 
> First, in my opinion it is *vital* that no special client is required to
> use Dispatch Router effectively. Transparency is a key property. You
> should be able to point a JMS application or qpid::messaging or any
> other compliant AMQP client at it. Any special coupling between this and
> proton based clients would in my view be a horrendous mistake. (Nothing
> like that is planned I'm sure, I'm just using this as a hypothetical
> example to make the point).
> 
> Second, the c

Re: Qpid Dispatch Router component

[Ted Ross]

On 10/11/2013 08:47 AM, Gordon Sim wrote:

On 10/11/2013 01:32 PM, Ted Ross wrote:

I have no love for the name.  But it has to be called *something* :)


Indeed, and I reiterate how hard good naming is.

Assuming 'Qpid Router' is undesirable as either making to big a claim 
(there will never be any other Qpid thing that routes) or provides 
insufficient 'identity', one option might be to follow the elementary 
particles approach started with proton. How about 'gluon' or perhaps 
'graviton'? :-)


I personally think a simple DX or QDX as an opaque name would be 
better as it more obviously acknowledges that its merely an identifier 
with little or no underlying description.


QDX is actually pretty good.  If we made that change, it wouldn't 
require significant file edits.




Again, this should in no way imply any opposition to proceeding as is. 
It is just an observation/suggestion.


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Gordon Sim]

On 10/11/2013 01:32 PM, Ted Ross wrote:

I have no love for the name.  But it has to be called *something* :)


Indeed, and I reiterate how hard good naming is.

Assuming 'Qpid Router' is undesirable as either making to big a claim 
(there will never be any other Qpid thing that routes) or provides 
insufficient 'identity', one option might be to follow the elementary 
particles approach started with proton. How about 'gluon' or perhaps 
'graviton'? :-)


I personally think a simple DX or QDX as an opaque name would be better 
as it more obviously acknowledges that its merely an identifier with 
little or no underlying description.


Again, this should in no way imply any opposition to proceeding as is. 
It is just an observation/suggestion.


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Ted Ross]

I have no love for the name.  But it has to be called *something* :)

On 10/11/2013 07:29 AM, Gordon Sim wrote:
One other point, I do think 'Dispatch' is a poor name. It is neither 
distinctive nor descriptive.


Appending 'Router' to it only underlines this in my view. That at 
least begins to explain what the component does. However 'Dispatch' 
then adds very little else.


That said I know how hard naming can be and this isn't an objection to 
making progress, simply an observation.



-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Ted Ross]

On 10/11/2013 07:19 AM, Gordon Sim wrote:

On 10/11/2013 11:52 AM, Rob Godfrey wrote:

On 11 October 2013 11:54, Gordon Sim  wrote:




Second, the code in Dispatch Router is in theory designed around a 
toolkit
for building  AMQP 'containers' of different kinds, with the router 
being
one such example (another might be a proxy focused more on enforcing 
ACLs
at the edge). In theory this could be viewed as an API of sorts. 
However I

think at this point its better viewed as a sensible desire for some
internal structure and separation of concerns. A publishable 'API' 
is in my
view some way off and would require a lot of work that would at this 
point
distract from the main goal, which is to define the behaviour of the 
router

and implement it.


This would be really cool... though I would hope/think this would 
lead to a
spin-off new component for the toolkit rather than being part of 
Dispatch

itself?


That would make sense to me, though I think we should cross that 
bridge if and when we ever come to it. (E.g. if a new component 
emerges that wants to reuse some of the same codebase).



 As such (and for the reasons you also allude to) I would think
this would not be a goal of Dispatch itself, but rather some sort of 
stated

aim and guiding principle of the development.


To me, the focus has to be on demonstrating the vision (and proving it 
works and has value). The rest is just good programming practice 
(modular design, with minimal coupling and maximum cohesion).



 Going off-topic a bit I
guess... but would we see such a framework as having multiple
implementations (in different languages) or only in C?


Again, to me the language used is secondary to the goal of proving the 
concept with real, deployable software. I do think that the 
communication patterns on top of AMQP between routers should be very 
clearly spelled out to allow clones or alternative implementations 
following the same rules to be developed if anyone anywhere has the 
desire to do so.


+1 on all of the above.

I'll add one more motivation/criteria for Dispatch.  Performance is a 
primary goal.  I've gone to some lengths to avoid bottlenecks in 
multi-threaded memory management and buffer/field management.  I believe 
that if Qpid has a flexible and scalable interconnect that does not 
introduce performance slowdowns, there will be many very interesting 
things that we can layer over it.


-Ted




-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Gordon Sim]

On 10/11/2013 12:27 AM, Steve Huston wrote:

The argument for evolving "de facto" standards is not really
pertinent here (in the context of addressing and management). De
facto standards emerge when some product/idea is developed and turned
loose and people take off and run with it. In this case, work is
ongoing at OASIS and other products are (I assume) implementing
them.


One of the biggest lessons I've learned during the years of AMQP's 
painful evolution, is that with standards, the technical details are not 
the only important thing. Indeed often they aren't even the most 
important thing. For want of a better word, 'social' aspects are critical.


For me, this is the area where AMQP made the biggest mistakes. Now that 
1.0 is here and is becoming a reality, I really hoped the same mistakes 
would not be perpetuated. AMQP has a history of mandating things that 
no-one implements or that are implemented grudgingly. It has a history 
of willing participants feeling excluded. (The dearth of practical 
choice for 1.0 clients is still a very real problem).


As I said before, my interest is not in reforming OASIS. My interest is 
in the Qpid project and in demonstrable interoperability and choice. I 
believe open source works best when software is produced by developers 
who want to do it, who feel personally engaged in and excited about what 
they are implementing, not when they feel coerced by outside forces.


The AMQP core specification is here and I fully accept its authority. 
The only 'authority' I recognise beyond that is the Qpid community and 
evidence of consensus with other products, particularly other open 
source projects.


A key principle at Apache is the 'faithful implementation of standards'. 
I firmly believe in that. The standards we do adopt we should implement 
faithfully. But simply because Qpid was founded to implement and promote 
AMQP does not mean that any additional 'emerging standard' being worked 
on under that name has special authority over the space it stakes out 
for itself.


That is not to say that I think they should be ignored. If there is no 
good reason not to adopt them, doing so makes sense. However I 
personally do not accept the notion that I have any obligation to engage 
with the OASIS process to address whatever issues I may have. My 
obligation is to discuss my views first and foremost with the Qpid 
community and the users of whatever software I am working on. If the 
view of the community is that only the 'emerging standard' can be 
implemented, so be it. It is the authority of the community I would 
respect there, not that of OASIS.


I stress again that I *want* genuine interoperability and real choice 
for users. I'm just not prepared to restrict myself to an approach and 
process I have lost faith in.


This is my own *personal* viewpoint and does not in anyway reflect the 
views of my employer or that of any colleague, friend or associate!


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Gordon Sim]

One other point, I do think 'Dispatch' is a poor name. It is neither 
distinctive nor descriptive.


Appending 'Router' to it only underlines this in my view. That at least 
begins to explain what the component does. However 'Dispatch' then adds 
very little else.


That said I know how hard naming can be and this isn't an objection to 
making progress, simply an observation.



-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey]

On 11 October 2013 13:19, Gordon Sim  wrote:

> On 10/11/2013 11:52 AM, Rob Godfrey wrote:
>
>> On 11 October 2013 11:54, Gordon Sim  wrote:
>>
>>
>>>
>>> Second, the code in Dispatch Router is in theory designed around a
>>> toolkit
>>> for building  AMQP 'containers' of different kinds, with the router being
>>> one such example (another might be a proxy focused more on enforcing ACLs
>>> at the edge). In theory this could be viewed as an API of sorts. However
>>> I
>>> think at this point its better viewed as a sensible desire for some
>>> internal structure and separation of concerns. A publishable 'API' is in
>>> my
>>> view some way off and would require a lot of work that would at this
>>> point
>>> distract from the main goal, which is to define the behaviour of the
>>> router
>>> and implement it.
>>>
>>>
>>>  This would be really cool... though I would hope/think this would lead
>> to a
>> spin-off new component for the toolkit rather than being part of Dispatch
>> itself?
>>
>
> That would make sense to me, though I think we should cross that bridge if
> and when we ever come to it. (E.g. if a new component emerges that wants to
> reuse some of the same codebase).
>
>
Yeah - that makes sense.


>
>   As such (and for the reasons you also allude to) I would think
>> this would not be a goal of Dispatch itself, but rather some sort of
>> stated
>> aim and guiding principle of the development.
>>
>
> To me, the focus has to be on demonstrating the vision (and proving it
> works and has value). The rest is just good programming practice (modular
> design, with minimal coupling and maximum cohesion).
>
>
+1


>
>   Going off-topic a bit I
>> guess... but would we see such a framework as having multiple
>> implementations (in different languages) or only in C?
>>
>
> Again, to me the language used is secondary to the goal of proving the
> concept with real, deployable software. I do think that the communication
> patterns on top of AMQP between routers should be very clearly spelled out
> to allow clones or alternative implementations following the same rules to
> be developed if anyone anywhere has the desire to do so.
>
>
>
Agreed on all the above :)

-- Rob


>
> --**--**-
> To unsubscribe, e-mail: 
> users-unsubscribe@qpid.apache.**org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>

Re: Qpid Dispatch Router component

[Gordon Sim]

On 10/11/2013 11:52 AM, Rob Godfrey wrote:

On 11 October 2013 11:54, Gordon Sim  wrote:




Second, the code in Dispatch Router is in theory designed around a toolkit
for building  AMQP 'containers' of different kinds, with the router being
one such example (another might be a proxy focused more on enforcing ACLs
at the edge). In theory this could be viewed as an API of sorts. However I
think at this point its better viewed as a sensible desire for some
internal structure and separation of concerns. A publishable 'API' is in my
view some way off and would require a lot of work that would at this point
distract from the main goal, which is to define the behaviour of the router
and implement it.



This would be really cool... though I would hope/think this would lead to a
spin-off new component for the toolkit rather than being part of Dispatch
itself?


That would make sense to me, though I think we should cross that bridge 
if and when we ever come to it. (E.g. if a new component emerges that 
wants to reuse some of the same codebase).



 As such (and for the reasons you also allude to) I would think
this would not be a goal of Dispatch itself, but rather some sort of stated
aim and guiding principle of the development.


To me, the focus has to be on demonstrating the vision (and proving it 
works and has value). The rest is just good programming practice 
(modular design, with minimal coupling and maximum cohesion).



 Going off-topic a bit I
guess... but would we see such a framework as having multiple
implementations (in different languages) or only in C?


Again, to me the language used is secondary to the goal of proving the 
concept with real, deployable software. I do think that the 
communication patterns on top of AMQP between routers should be very 
clearly spelled out to allow clones or alternative implementations 
following the same rules to be developed if anyone anywhere has the 
desire to do so.



-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey]

On 11 October 2013 11:54, Gordon Sim  wrote:

>
>
> Second, the code in Dispatch Router is in theory designed around a toolkit
> for building  AMQP 'containers' of different kinds, with the router being
> one such example (another might be a proxy focused more on enforcing ACLs
> at the edge). In theory this could be viewed as an API of sorts. However I
> think at this point its better viewed as a sensible desire for some
> internal structure and separation of concerns. A publishable 'API' is in my
> view some way off and would require a lot of work that would at this point
> distract from the main goal, which is to define the behaviour of the router
> and implement it.
>
>
This would be really cool... though I would hope/think this would lead to a
spin-off new component for the toolkit rather than being part of Dispatch
itself?  As such (and for the reasons you also allude to) I would think
this would not be a goal of Dispatch itself, but rather some sort of stated
aim and guiding principle of the development.  Going off-topic a bit I
guess... but would we see such a framework as having multiple
implementations (in different languages) or only in C?

-- Rob


>
> --**--**-
> To unsubscribe, e-mail: 
> users-unsubscribe@qpid.apache.**org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>

Re: Qpid Dispatch Router component

[Rob Godfrey]

On 10 October 2013 20:07, Ted Ross  wrote:

>
> On 10/10/2013 11:20 AM, Rob Godfrey wrote:
>
>> On 10 October 2013 15:35, Ted Ross  wrote:
>>
>>  On 10/10/2013 04:38 AM, Rob Godfrey wrote:
>>>
>>>
>
[...]


> So here's the root of the issue.  The question is whether or not Dispatch
> is too tangential in its relation to AMQP to be considered appropriate to
> be hosted in Apache Qpid.
>
>
To me that's *really* not the root of the issue at all.  The issue to me is
that I don't know what Dispatch is, how it differs in scope and vision from
a broker or other component, and how it fits into the Qpid story.  In
practice I probably do know because of off-list conversations, but I
couldn't get that information from the mails or documentation here.  I
think if we were to include the definition/description that Gordon gave
then we would have a really good scope statement on what it was about.


> You stated later in your email that you don't know what an AMQP-compliant
> router is.  Well, nobody really does because nothing like it has ever
> existed.  The very idea was not even conceivable before there was an AMQP
> 1.0.  But now we have AMQP and a door has been opened to create really
> interesting solutions to difficult problems in large-scale distributed
> computing. Dispatch is an early attempt to implement such a solution.  If
> we decide that by "tangential" we mean "beyond the scope of traditional
> broker-based messaging", then Dispatch is clearly tangential and I will go
> and find another community to host it.
>

Again this is really not what I'm saying.  As Fraser's mail indicates there
is currently no clarity in how the various Qpid components fit together.
What I want is for us to improve that by having a clear and *shared*
understanding of what all the components are.


>
> The only way to know if a project will have long-term value is to let it
> run for awhile.  It's a form of incubation.  We need a way, as you suggest,
> of deciding what to incubate.  We then need a way to decide when a
> sub-project has failed and should be abandoned. Given that we have no such
> guidelines in place, I intend to go forward with the same process we used
> for the JMS project.  If there arises a consensus in the community that
> Dispatch does not, or might not belong in Qpid, I will not go forward.
>
> In the mean time, I will try to fill in some of the gaps in information
> that you've identified.
>
>
So, to reiterate I am in no way suggesting that Dispatch shouldn't be a
component in Qpid... I actually think it sounds like a really good idea and
something we should be doing.  However I think we should be aiming at
better explaining to ourselves and others what the goals behind our
components are.

i do think we were remiss in not setting up a proper scope statement for
the JMS client.  I'd actually suggest that we come up with one now and put
it to a vote to ensure that we have broad agreement on the proposed
component and to discover now if there are any misconceptions/disagreements
about the form that work should take.  I'll see if Robbie and I can come up
with something on that one (though obviously others are welcome to do the
work there instead :-) ).

Cheers,
Rob


> -Ted
>
>
>
> --**--**-
> To unsubscribe, e-mail: 
> users-unsubscribe@qpid.apache.**org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>

Re: Qpid Dispatch Router component

[Rob Godfrey]

So, just to be clear the reason I asked the question is to try to help us
frame better a scope statement.

People often ask why we currently have both a Java Broker and a C++ broker,
and I think it's a fair question :-).  If we start building up Dispatch as
a component and someone says "hey - it'd be really cool to have some sort
of message store built into Dispatch" and so on... how do we define the
difference between what Dispatch might become and what we have in brokers
currently?  The questions on client APIs were really a nod to the confusion
in scope that I see in Proton.  Again, within Messenger I see it as not a
huge step to suddenly add some storage and a bit of configuration and again
... you have something that looks a lot like a broker.  Going back to
Fraser's mail I think we need to be clear on what each component is and
what it isn't (being clear on it ourselves seems to be a pre-requisite of
being able to explain it to others :-) ).

-- Rob

On 11 October 2013 11:54, Gordon Sim  wrote:

> I don't claim to speak with any authority on Dispatch Router, but fwiw,
> here's my answer to these two specific questions:
>
>
> On 10/10/2013 04:20 PM, Rob Godfrey wrote:
>
>> How does a router differ from a broker?
>>
>
> I tend to have a more general view of what 'broker' might mean, but in the
> context of statements that Dispatch Router 'is not a broker', what is meant
> is the type of intermediary as exemplified by the two existing Qpid brokers
> and others like them.
>
> As I see it, there are two key differences between Dispatch Router and
> these.
>
> The first is that a principle of the Dispatch Router is that it does not
> 'take ownership' of messages. This is in contrast to most brokers where the
> publisher transfers responsibility for messages to the broker and the
> broker then undertakes to deliver them allowing the producer to forget
> about them. With Dispatch Router, the idea is that it will route those
> messages, but any guarantee regarding acceptance of those messages comes
> from the consumer(s) of the message, not from the intermediary.
>
> The second distinction is that Dispatch Router is from the start designed
> to be a small piece in a distributed routing network of interconnected
> components.
>
> Now, there is nothing that says a broker can't do these things. In fact I
> added code a while back to the Qpid c++ broker that does the former -
> relays messages through it, providing end-to-end guarantees.
>
> Likewise the distributed architecture has its roots in things like the
> federation in the Qpid c++ broker (and similar solutions by other brokers).
>
> The Router however is setting out from the start to do *only* these
> things. It is based on the idea that these are separate concerns that can
> be split apart allowing 'brokers' to focus on queueing, persistence,
> transactions, augmented functionality such as LVQ patterns etc, and that
> the 'federation' logic can be provided in a separate component that can
> hook up different brokers into a federation as well as hooking in other
> types of AMQP based services and allow clients to connect directly to the
> router as well.
>
> By focusing on one aspect its thought components can do a better job. E.g.
> as I understand it, the Dispatch Router aims to address some of the
> weaknesses of the federation support in qpidd, by providing for redundant,
> fault-tolerant routes without duplication of messages, adapting to failures
> and dynamically computing the best path between two endpoints.
>
> I confess, for all its faults, federation has been one of my favourite
> features of the c++ broker because of the potential I see in it. That is
> why I started off adding some 1.0 related improvements. However I think
> Ted's thinking on Dispatch Router is quite compelling (though I also feel a
> lot of the necessary detail is still missing and it still needs to be
> proven).
>
> For my part, I am excited to see how this develops, am keen to take part
> if I can and would encourage anyone with an interest to chip in with ideas,
> feedback, criticism, questions, code, use cases or whatever else. This
> began as Ted's 'brainchild' - and most things begin as someone's idea - but
> it really is now open to anyone and everyone who is interested. Get
> involved and shape the direction!
>
>
>  Would you expect any special client APIs to form part of the router
>> package or not?
>>
>
> I'm not entirely sure if I understand the question, but there are again
> two points I would make.
>
> First, in my opinion it is *vital* that no special client is required to
> use Dispatch Router effectively. Transparency is a key property. You should
> be able to point a JMS application or qpid::messaging or any other
> compliant AMQP client at it. Any special coupling between this and proton
> based clients would in my view be a horrendous mistake. (Nothing like that
> is planned I'm sure, I'm just using this as a hypothetical example to make
> the point).
>

Re: Qpid Dispatch Router component

[Gordon Sim]

I don't claim to speak with any authority on Dispatch Router, but fwiw,
here's my answer to these two specific questions:

On 10/10/2013 04:20 PM, Rob Godfrey wrote:

How does a router differ from a broker?


I tend to have a more general view of what 'broker' might mean, but in 
the context of statements that Dispatch Router 'is not a broker', what 
is meant is the type of intermediary as exemplified by the two existing 
Qpid brokers and others like them.


As I see it, there are two key differences between Dispatch Router and 
these.


The first is that a principle of the Dispatch Router is that it does not 
'take ownership' of messages. This is in contrast to most brokers where 
the publisher transfers responsibility for messages to the broker and 
the broker then undertakes to deliver them allowing the producer to 
forget about them. With Dispatch Router, the idea is that it will route 
those messages, but any guarantee regarding acceptance of those messages 
comes from the consumer(s) of the message, not from the intermediary.


The second distinction is that Dispatch Router is from the start 
designed to be a small piece in a distributed routing network of 
interconnected components.


Now, there is nothing that says a broker can't do these things. In fact 
I added code a while back to the Qpid c++ broker that does the former - 
relays messages through it, providing end-to-end guarantees.


Likewise the distributed architecture has its roots in things like the 
federation in the Qpid c++ broker (and similar solutions by other brokers).


The Router however is setting out from the start to do *only* these 
things. It is based on the idea that these are separate concerns that 
can be split apart allowing 'brokers' to focus on queueing, persistence, 
transactions, augmented functionality such as LVQ patterns etc, and that 
the 'federation' logic can be provided in a separate component that can 
hook up different brokers into a federation as well as hooking in other 
types of AMQP based services and allow clients to connect directly to 
the router as well.


By focusing on one aspect its thought components can do a better job. 
E.g. as I understand it, the Dispatch Router aims to address some of the 
weaknesses of the federation support in qpidd, by providing for 
redundant, fault-tolerant routes without duplication of messages, 
adapting to failures and dynamically computing the best path between two 
endpoints.


I confess, for all its faults, federation has been one of my favourite 
features of the c++ broker because of the potential I see in it. That is 
why I started off adding some 1.0 related improvements. However I think 
Ted's thinking on Dispatch Router is quite compelling (though I also 
feel a lot of the necessary detail is still missing and it still needs 
to be proven).


For my part, I am excited to see how this develops, am keen to take part 
if I can and would encourage anyone with an interest to chip in with 
ideas, feedback, criticism, questions, code, use cases or whatever else. 
This began as Ted's 'brainchild' - and most things begin as someone's 
idea - but it really is now open to anyone and everyone who is 
interested. Get involved and shape the direction!



Would you expect any special client APIs to form part of the router
package or not?


I'm not entirely sure if I understand the question, but there are again 
two points I would make.


First, in my opinion it is *vital* that no special client is required to 
use Dispatch Router effectively. Transparency is a key property. You 
should be able to point a JMS application or qpid::messaging or any 
other compliant AMQP client at it. Any special coupling between this and 
proton based clients would in my view be a horrendous mistake. (Nothing 
like that is planned I'm sure, I'm just using this as a hypothetical 
example to make the point).


Second, the code in Dispatch Router is in theory designed around a 
toolkit for building  AMQP 'containers' of different kinds, with the 
router being one such example (another might be a proxy focused more on 
enforcing ACLs at the edge). In theory this could be viewed as an API of 
sorts. However I think at this point its better viewed as a sensible 
desire for some internal structure and separation of concerns. A 
publishable 'API' is in my view some way off and would require a lot of 
work that would at this point distract from the main goal, which is to 
define the behaviour of the router and implement it.


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

RE: Qpid Dispatch Router component

[Steve Huston]

Hi guys,

> -Original Message-
> From: Rob Godfrey [mailto:rob.j.godf...@gmail.com]
> Sent: Thursday, October 10, 2013 11:21 AM
> To: users@qpid.apache.org
> Subject: Re: Qpid Dispatch Router component
> 
> On 10 October 2013 15:35, Ted Ross  wrote:
> 
> >
> > On 10/10/2013 04:38 AM, Rob Godfrey wrote:
> >
> >> My main concern is that I believe that Qpid should be primarily
> >> directed at implementing AMQP standards, and building resuable
> >> toolkits and components that fit into any AMQP network. I'd be very
> >> concerned if we were inventing alternative management protocols, or
> >> building components that only interoperated with other Qpid tools.
> >> So, personally I'd like to see statement around components saying
> >> that they will be fully implementing emerging AMQP Management,
> AMQP
> >> addressing, etc. standards, and that we as a project then ensure we stick
> to these goals.
> >>
> >
> > Rob,
> >
> > Here's where I'm going to have to disagree with you in principle. I
> > believe that Qpid should be primarily directed at innovating with AMQP
> > and helping to drive the AMQP standards where appropriate.  If Qpid
> > doesn't do it, somebody else will.  I should point out that almost
> > everything we do here is well ahead of the standards, including the JMS
> client.
> >
> > The thing I object to in your statement is the direction of flow from
> > the standard to the implementation.  Standards bodies _do not
> > innovate_.  If the emerging standards are such that a particular
> > valuable innovation cannot be done, the standard needs to change or be
> > ignored.  Qpid must not allow itself to be put in the position of
> > meekly sitting and waiting for the tablets to come from on high before
> implementing.
> >
> >
> I'm not saying that there is a direction of standard -> implementation, but
> that if there is a standard we should be implementing it rather than rolling
> our own which conflicts or substantially overlaps.  If we see a standard
> emerging at OASIS and do not believe it is correct then we should be working
> to fix it at that venue.  If we do work which we think is generally 
> applicable to
> other AMQP implementations then we should be considering whether we
> want to push it as a standard at OASiS or elsewhere.

The above is, as I see it, the crux of the matter. I agree with Rob on this 
item. Addressing and management are evolving standards in OASIS and they should 
not be ignored. The argument for evolving "de facto" standards is not really 
pertinent here (in the context of addressing and management). De facto 
standards emerge when some product/idea is developed and turned loose and 
people take off and run with it. In this case, work is ongoing at OASIS and 
other products are (I assume) implementing them. Ignoring that will only invite 
difficulties down the road, especially if Dispatch ends up only able to talk to 
itself. And I REALLY, REALLY want Dispatch to take off - I have big ideas for 
its use already.

As far as the idea of a AMQP router, now that's an opportunity for de facto 
standard(s). As long as it interoperates with any other AMQP 1.0 endpoint that 
uses AMQP standard addressing and it responds correctly to AMQP standard 
management.

Picture yourself as in Cisco's shoes 25-30 years ago. Take it from that 
approach and you'll be golden.

> I absolutely don't think that Qpid should be restricted in scope to simply
> implementing the standards, and I strongly believe that Qpid can be a place
> where we innovate and then work towards bringing behaviours to a
> standards body.  However I also think that when we do so we should state
> up front what it is we are trying to achieve.  I also don't think that qpid 
> should
> become a mini-GitHub for any project that is tangentially related to AMQP to
> simply use as a code repository.
> 
> So, in the case of Dispatch, it certainly seems to include innovative ideas
> which do not form part of any AMQP standard (proposed or current) but also
> seems to hint at overlaps with such (emerging) standards in addressing and
> in management.  In addressing I think it's important that any requirements
> for addressing that you believe are important for Dispatch are discussed and
> considered in the OASIS AMQP work... in the Management space I would
> very much hope that the emerging Management spec for AMQP would
> prove to be a foundation on which functionality could be built and I would be
> concerned for a number of reasons if you felt that Dispatch
> couldn't/shouldn't be using them.

Absolutely.

-Steve


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Ted Ross]

On 10/10/2013 11:20 AM, Rob Godfrey wrote:

On 10 October 2013 15:35, Ted Ross  wrote:


On 10/10/2013 04:38 AM, Rob Godfrey wrote:


My main concern is that I believe that Qpid should be primarily directed
at implementing AMQP standards, and building resuable toolkits and
components that fit into any AMQP network. I'd be very concerned if we were
inventing alternative management protocols, or building components that
only interoperated with other Qpid tools. So, personally I'd like to see
statement around components saying that they will be fully implementing
emerging AMQP Management, AMQP addressing, etc. standards, and that we as a
project then ensure we stick to these goals.


Rob,

Here's where I'm going to have to disagree with you in principle. I
believe that Qpid should be primarily directed at innovating with AMQP and
helping to drive the AMQP standards where appropriate.  If Qpid doesn't do
it, somebody else will.  I should point out that almost everything we do
here is well ahead of the standards, including the JMS client.

The thing I object to in your statement is the direction of flow from the
standard to the implementation.  Standards bodies _do not innovate_.  If
the emerging standards are such that a particular valuable innovation
cannot be done, the standard needs to change or be ignored.  Qpid must not
allow itself to be put in the position of meekly sitting and waiting for
the tablets to come from on high before implementing.



I'm not saying that there is a direction of standard -> implementation, but
that if there is a standard we should be implementing it rather than
rolling our own which conflicts or substantially overlaps.  If we see a
standard emerging at OASIS and do not believe it is correct then we should
be working to fix it at that venue.  If we do work which we think is
generally applicable to other AMQP implementations then we should be
considering whether we want to push it as a standard at OASiS or elsewhere.

I absolutely don't think that Qpid should be restricted in scope to simply
implementing the standards, and I strongly believe that Qpid can be a place
where we innovate and then work towards bringing behaviours to a standards
body.  However I also think that when we do so we should state up front
what it is we are trying to achieve.  I also don't think that qpid should
become a mini-GitHub for any project that is tangentially related to AMQP
to simply use as a code repository.


So here's the root of the issue.  The question is whether or not 
Dispatch is too tangential in its relation to AMQP to be considered 
appropriate to be hosted in Apache Qpid.


You stated later in your email that you don't know what an 
AMQP-compliant router is.  Well, nobody really does because nothing like 
it has ever existed.  The very idea was not even conceivable before 
there was an AMQP 1.0.  But now we have AMQP and a door has been opened 
to create really interesting solutions to difficult problems in 
large-scale distributed computing. Dispatch is an early attempt to 
implement such a solution.  If we decide that by "tangential" we mean 
"beyond the scope of traditional broker-based messaging", then Dispatch 
is clearly tangential and I will go and find another community to host it.


The only way to know if a project will have long-term value is to let it 
run for awhile.  It's a form of incubation.  We need a way, as you 
suggest, of deciding what to incubate.  We then need a way to decide 
when a sub-project has failed and should be abandoned. Given that we 
have no such guidelines in place, I intend to go forward with the same 
process we used for the JMS project.  If there arises a consensus in the 
community that Dispatch does not, or might not belong in Qpid, I will 
not go forward.


In the mean time, I will try to fill in some of the gaps in information
that you've identified.

-Ted



-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey]

On 10 October 2013 15:35, Ted Ross  wrote:

>
> On 10/10/2013 04:38 AM, Rob Godfrey wrote:
>
>> My main concern is that I believe that Qpid should be primarily directed
>> at implementing AMQP standards, and building resuable toolkits and
>> components that fit into any AMQP network. I'd be very concerned if we were
>> inventing alternative management protocols, or building components that
>> only interoperated with other Qpid tools. So, personally I'd like to see
>> statement around components saying that they will be fully implementing
>> emerging AMQP Management, AMQP addressing, etc. standards, and that we as a
>> project then ensure we stick to these goals.
>>
>
> Rob,
>
> Here's where I'm going to have to disagree with you in principle. I
> believe that Qpid should be primarily directed at innovating with AMQP and
> helping to drive the AMQP standards where appropriate.  If Qpid doesn't do
> it, somebody else will.  I should point out that almost everything we do
> here is well ahead of the standards, including the JMS client.
>
> The thing I object to in your statement is the direction of flow from the
> standard to the implementation.  Standards bodies _do not innovate_.  If
> the emerging standards are such that a particular valuable innovation
> cannot be done, the standard needs to change or be ignored.  Qpid must not
> allow itself to be put in the position of meekly sitting and waiting for
> the tablets to come from on high before implementing.
>
>
I'm not saying that there is a direction of standard -> implementation, but
that if there is a standard we should be implementing it rather than
rolling our own which conflicts or substantially overlaps.  If we see a
standard emerging at OASIS and do not believe it is correct then we should
be working to fix it at that venue.  If we do work which we think is
generally applicable to other AMQP implementations then we should be
considering whether we want to push it as a standard at OASiS or elsewhere.

I absolutely don't think that Qpid should be restricted in scope to simply
implementing the standards, and I strongly believe that Qpid can be a place
where we innovate and then work towards bringing behaviours to a standards
body.  However I also think that when we do so we should state up front
what it is we are trying to achieve.  I also don't think that qpid should
become a mini-GitHub for any project that is tangentially related to AMQP
to simply use as a code repository.

So, in the case of Dispatch, it certainly seems to include innovative ideas
which do not form part of any AMQP standard (proposed or current) but also
seems to hint at overlaps with such (emerging) standards in addressing and
in management.  In addressing I think it's important that any requirements
for addressing that you believe are important for Dispatch are discussed
and considered in the OASIS AMQP work... in the Management space I would
very much hope that the emerging Management spec for AMQP would prove to be
a foundation on which functionality could be built and I would be concerned
for a number of reasons if you felt that Dispatch couldn't/shouldn't be
using them.


> So here's my proposed statement regarding Dispatch:
>
> Qpid Dispatch is an implementation of an AMQP-compliant router. Dispatch
> is pursuing a specific approach to routing and addressing that may differ
> from other approaches.  The implementation will conform to all relevant
> emerging standards (Management, Addressing, and Security) to the extent
> that it is practical.  In the event that there are parts of the
> specifications that are not practical to implement, we shall provide
> specifics to the standards committee in an effort to improve the
> specifications.
>
>
Around AMQP compliance that seems reasonable, but from the rest of that
statement I'm not sure what "AMQP-compliant router" means... How does a
router differ from a broker?  Would you expect any special client APIs to
form part of the router package or not?  Would you expect any other
implementations of Dispatch, or is it intended only to be in C?  Is it
intended to be embedded in other programs or always to act as a stand-alone
executable?

What I'm getting at with a scope statement is that we should be clear at
the outset of projects what we are looking to achieve.  We may deliberately
leave thing very open, but then we should state that clearly too.

In order to be able to have a clear story about how the components of Qpid
fit together I think we should be clear before we set out on the journey of
a component what we are looking to do (and what we are explicitly not
looking to do).  For previous components I don't think we've done that well
enough (and for the upcoming JMS client I want to go back and rectify that
omission).

-- Rob

-Ted
>
>
> --**--**-
>
> To unsubscribe, e-mail: 
> users-unsubscribe@qpid.apache.**org
> For additional commands, e-mail: users-h...@qpid.

Re: Qpid Dispatch Router component

[Gordon Sim]

On 10/10/2013 12:09 PM, Rob Godfrey wrote:

On 10 October 2013 12:46, Gordon Sim  wrote:

On 10/10/2013 10:58 AM, Rob Godfrey wrote:

I think the point of Qpid (vs. any other messaging
implementation at Apache or elsewhere) is to implement the AMQP
specification.


I have no disagreement when the AMQP specification is what is currently
published.

My concern is where that is defined to be an expanding, all-encompassing
effort that ring fences whole areas of behaviour and sets itself up as the
only legitimate avenue for exploration.


Do you believe that is what the OASIS TCs are doing?  If so in which areas?


The idea of a general policy that any Qpid component must support any 
emerging standard from those TCs in a particular area and cannot explore 
alternatives or anything that may overlap does feel very much like this.


I am not saying that is what you meant, merely expressing that I would 
not be comfortable with such a policy if it was.



Expanding beyond the current specification needs a more diverse source of
ideas and a more open, transparent and collaborative process.


While I agree that a more open process is most definitely desirable, I
would argue that the OASIS TCs (sadly) currently have a much more diverse
membership than Qpid committers.Also note that you do not need to be a
member of OASIS to comment on work going on there.  If you (or anyone else)
believe that the work going on in OASIS is misguided you should comment
there [1], not here.


While involved at OASIS I did comment. However, I am not now seeking to 
reform the AMQP work at OASIS. Any such effort should indeed take place 
there and not here.


Neither am I trying to subvert it. I have no issue with you or anyone 
else working on projects within Qpid related to specifications you are 
progressing at OASIS, providing anyone else can join in and contribute 
ideas.


All I am saying is that I don't consider OASIS to have authority over 
what other work goes on in Qpid, again provided that work is done in an 
open and collaborative manner which listens to criticism of the approach.


Collaboration doesn't have to mean that everyone agrees on one point of 
view. A degree of exploration of alternatives is in my view healthy at 
this point provided we all conform to the underlying specification as 
published and strive to ensure the various voices in the community are 
heard and that whatever is produced serves the needs of those using it.


I myself will very happily contribute to, align with and support any 
effort that I see emerging with genuine widespread support from existing 
communities. If that is a spec from OASIS, great. If it's an idea from 
one of the other implementers, great. I want any additional standard to 
be adopted on merit not by fiat.


In the end I believe we want the same thing, we have the same goal of 
richer interoperability and I do firmly believe that while in theory we 
have different views on how best to achieve that we can collaborate 
effectively on the practical side of aiding AMQP adoption and on 
concrete issues we are as often as not likely to find a happy compromise.


Now, as to improving the process at Qpid and building a more diverse set 
of contributors, that I am very much interested in and this is the 
perfect place to discuss it. I would be eager to hear from anyone with 
ideas that could help.



While I may not be entirely happy with the OASIS process, the value in AMQP
is in standardisation.


The value is in practical interoperability and interchangeability.

Though I personally don't think OASIS is the best route to achieving 
that, I respect your view that it is. There should be room for both of 
us to help users reap the benefits that AMQP promised.



 If Qpid can be a place where multiple parties can
come together and work together then it may be a good place for de-facto
standards to emerge.


I want to be really clear on this point as it comes up several times 
here and in previous mails.


I said that I believe the emergence of de-facto standards is something 
to be nurtured and encouraged. I think Qpid can have a part in that, but 
I think it can *only* be a part. It *must* involve other communities, 
projects and products. So I am certainly *not* suggesting that I believe 
that work at Qpid should be taken as a de-facto standard.


[...]

Historically Qpid has not been seen as a neutral place.  For better or
worse, there are a preponderance of committers from one organisation.  In
order to counter perceptions of a lack of neutrality I think work has to be
done to bring more people in before we try to sell ourselves as a neutral
venue for emerging standards.


Again, just so there is no misunderstanding, I am *not* 'selling Qpid' 
as such a venue...



 Until we can actually demonstrate that Qpid
is such a place I think it is presumptuous of us to try to claim de-facto
standards for our work.


...and I have made no such claim.

A de-facto standard is not something you ca

Re: Qpid Dispatch Router component

[Ted Ross]

On 10/10/2013 04:38 AM, Rob Godfrey wrote:
My main concern is that I believe that Qpid should be primarily 
directed at implementing AMQP standards, and building resuable 
toolkits and components that fit into any AMQP network. I'd be very 
concerned if we were inventing alternative management protocols, or 
building components that only interoperated with other Qpid tools. So, 
personally I'd like to see statement around components saying that 
they will be fully implementing emerging AMQP Management, AMQP 
addressing, etc. standards, and that we as a project then ensure we 
stick to these goals.


Rob,

Here's where I'm going to have to disagree with you in principle. I 
believe that Qpid should be primarily directed at innovating with AMQP 
and helping to drive the AMQP standards where appropriate.  If Qpid 
doesn't do it, somebody else will.  I should point out that almost 
everything we do here is well ahead of the standards, including the JMS 
client.


The thing I object to in your statement is the direction of flow from 
the standard to the implementation.  Standards bodies _do not 
innovate_.  If the emerging standards are such that a particular 
valuable innovation cannot be done, the standard needs to change or be 
ignored.  Qpid must not allow itself to be put in the position of meekly 
sitting and waiting for the tablets to come from on high before 
implementing.


So here's my proposed statement regarding Dispatch:

Qpid Dispatch is an implementation of an AMQP-compliant router. Dispatch 
is pursuing a specific approach to routing and addressing that may 
differ from other approaches.  The implementation will conform to all 
relevant emerging standards (Management, Addressing, and Security) to 
the extent that it is practical.  In the event that there are parts of 
the specifications that are not practical to implement, we shall provide 
specifics to the standards committee in an effort to improve the 
specifications.


-Ted


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey]

On 10 October 2013 12:46, Gordon Sim  wrote:

> On 10/10/2013 10:58 AM, Rob Godfrey wrote:
>
>> I think the point of Qpid (vs. any other messaging
>> implementation at Apache or elsewhere) is to implement the AMQP
>> specification.
>>
>
> I have no disagreement when the AMQP specification is what is currently
> published.
>
> My concern is where that is defined to be an expanding, all-encompassing
> effort that ring fences whole areas of behaviour and sets itself up as the
> only legitimate avenue for exploration.
>

Do you believe that is what the OASIS TCs are doing?  If so in which areas?


>

Expanding beyond the current specification needs a more diverse source of
> ideas and a more open, transparent and collaborative process.
>
>
While I agree that a more open process is most definitely desirable, I
would argue that the OASIS TCs (sadly) currently have a much more diverse
membership than Qpid committers.  Also note that you do not need to be a
member of OASIS to comment on work going on there.  If you (or anyone else)
believe that the work going on in OASIS is misguided you should comment
there [1], not here.

While I may not be entirely happy with the OASIS process, the value in AMQP
is in standardisation.  If Qpid can be a place where multiple parties can
come together and work together then it may be a good place for de-facto
standards to emerge.


> [...]
>
>  I think that any such efforts at de-facto standardisation
>> must first reach out to other AMQP implementers and ensure there is a
>> broad
>> agreement on direction.  If the Qpid project can be a vehicle for doing
>> this, then great - however currently that is not how Qpid is operating and
>> I would be very concerned at us trying to claim any sort of work done
>> within Qpid as a "de-facto" standard.
>>
>
> Indeed and that of course is a straw man since no one is suggesting that
> we claim any such thing.
>
> However I would be very concerned if the OASIS TCs, with such limited
> representation, were enshrined as authorities over any and every aspect of
> work they choose to start.
>
> At present there is sadly no AMQP-wide community. While Qpid is far from
> perfect, it is at least, as an Apache project, founded on the  ethos of
> open, community driven collaboration. It has rules for governance that
> provide the means to correct deviations from that. Everything we do should
> be open and subject to debate and consensus.
>

Indeed... I would very much like to have an AMQP-wide community
established.  If Qpid can become the home of that then that is great,
however in order to facilitate that then I believe the first step would be
to be good AMQP citizens and work with and implement the emerging standards
that are currently being progressed (or be clear in our objections to said
work).

Historically Qpid has not been seen as a neutral place.  For better or
worse, there are a preponderance of committers from one organisation.  In
order to counter perceptions of a lack of neutrality I think work has to be
done to bring more people in before we try to sell ourselves as a neutral
venue for emerging standards.  Until we can actually demonstrate that Qpid
is such a place I think it is presumptuous of us to try to claim de-facto
standards for our work.  With greatest respect to those involved, that is
the way that leads to the next QMF.

I want users of Qpid to be assured that components or libraries that they
can get here will work interoperably with any AMQP implementation that
follows the standards.


>
> Reaching out to - and collaborating with - other implementations and
> communities is something I personally feel we must do more of in order to
> realise the promise behind AMQP. Starting those discussions here seems like
> one (though certainly not the only) reasonable way to begin however.
>
>
>
-- Rob

[1] https://www.oasis-open.org/committees/comments/index.php?wg_abbrev=amqp


> --**--**-
> To unsubscribe, e-mail: 
> users-unsubscribe@qpid.apache.**org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>

Re: Qpid Dispatch Router component

[Gordon Sim]

On 10/10/2013 10:58 AM, Rob Godfrey wrote:

I think the point of Qpid (vs. any other messaging
implementation at Apache or elsewhere) is to implement the AMQP
specification.


I have no disagreement when the AMQP specification is what is currently 
published.


My concern is where that is defined to be an expanding, all-encompassing 
effort that ring fences whole areas of behaviour and sets itself up as 
the only legitimate avenue for exploration.


Expanding beyond the current specification needs a more diverse source 
of ideas and a more open, transparent and collaborative process.


[...]

I think that any such efforts at de-facto standardisation
must first reach out to other AMQP implementers and ensure there is a broad
agreement on direction.  If the Qpid project can be a vehicle for doing
this, then great - however currently that is not how Qpid is operating and
I would be very concerned at us trying to claim any sort of work done
within Qpid as a "de-facto" standard.


Indeed and that of course is a straw man since no one is suggesting that 
we claim any such thing.


However I would be very concerned if the OASIS TCs, with such limited 
representation, were enshrined as authorities over any and every aspect 
of work they choose to start.


At present there is sadly no AMQP-wide community. While Qpid is far from 
perfect, it is at least, as an Apache project, founded on the  ethos of 
open, community driven collaboration. It has rules for governance that 
provide the means to correct deviations from that. Everything we do 
should be open and subject to debate and consensus.


Reaching out to - and collaborating with - other implementations and 
communities is something I personally feel we must do more of in order 
to realise the promise behind AMQP. Starting those discussions here 
seems like one (though certainly not the only) reasonable way to begin 
however.


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Rob Godfrey]

On 10 October 2013 11:49, Gordon Sim  wrote:

> On 10/10/2013 09:38 AM, Rob Godfrey wrote:
>
>> My main concern is that I believe that Qpid should be primarily directed
>> at
>> implementing AMQP standards, and building resuable toolkits and components
>> that fit into any AMQP network.  I'd be very concerned if we were
>> inventing
>> alternative management protocols, or building components that only
>> interoperated with other Qpid tools.
>>
>> So, personally I'd like to see statement around components saying that
>> they
>> will be fully implementing emerging AMQP Management, AMQP addressing, etc.
>> standards, and that we as a project then ensure we stick to these goals.
>>
>
> Interoperability with other AMQP compliant components both in and out of
> Qpid and Apache is certainly key. That is what AMQP is all about and what
> Qpid should be about.
>
> Faithful implementation of the existing AMQP specification is clearly a
> requirement as that is central to the charter of the project. Where any
> auxiliary or emerging specifications are adopted by a component, whether
> they are AMQP related or not, they should be compliant with that.
>
> However, having a general policy where all Qpid components are required to
> implement whatever 'emerging standards' come out of the OASIS AMQP TCs is
> not something I am comfortable with.
>

Saying that all Qpid components implement all emerging standards is clearly
not what I am saying, because not all standards are appropriate for all
components.  However I think the point of Qpid (vs. any other messaging
implementation at Apache or elsewhere) is to implement the AMQP
specification.  I'd generally question why work was being carried out in
Qpid (as opposed to elsewhere) if the work is not based on existing or
emerging AMQP standards.


>
> The Qpid community as a whole needs to have a say in how future features
> will work through an open, collaborative process (open to _anyone_, even
> those primarily involved with other AMQP related projects or products).
>
>
i completely agree.


> Personally I think this would be better for AMQP as well. Allowing and
> encouraging the emergence of grass-roots driven, de-facto 'standards'
> developed in and between open, collaborative and transparent communities
> and backed up by proven interoperable implementations, which can then form
> the basis for official de-jure standardisation.
>
>
Possibly, but I think that any such efforts at de-facto standardisation
must first reach out to other AMQP implementers and ensure there is a broad
agreement on direction.  If the Qpid project can be a vehicle for doing
this, then great - however currently that is not how Qpid is operating and
I would be very concerned at us trying to claim any sort of work done
within Qpid as a "de-facto" standard.  Where there is qork going on in AMQP
then we as the Qpid community should be ensuring that we feed back to that
and raise questions/objections as necessary (whether we are part of the
OASIS group or not - feedback from the public is possible).

-- Rob

--**--**-
> To unsubscribe, e-mail: 
> users-unsubscribe@qpid.apache.**org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>

Re: Qpid AMQP 1.0 - How does it all hang together? - was Re: Qpid Dispatch Router component

[Gordon Sim]

On 10/09/2013 08:04 PM, Ted Ross wrote:

There are a lot of new ideas floating around, some of them overlapping.
I think Qpid is a perfect place to explore and develop new technologies
based on AMQP.  This will cause some confusion and force us to work at
articulating what we are doing and thinking, and it will be people like
you who will prompt the important discussions.


Very well said, I heartily agree!

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Dispatch Router component

[Gordon Sim]

On 10/10/2013 09:38 AM, Rob Godfrey wrote:

My main concern is that I believe that Qpid should be primarily directed at
implementing AMQP standards, and building resuable toolkits and components
that fit into any AMQP network.  I'd be very concerned if we were inventing
alternative management protocols, or building components that only
interoperated with other Qpid tools.

So, personally I'd like to see statement around components saying that they
will be fully implementing emerging AMQP Management, AMQP addressing, etc.
standards, and that we as a project then ensure we stick to these goals.


Interoperability with other AMQP compliant components both in and out of 
Qpid and Apache is certainly key. That is what AMQP is all about and 
what Qpid should be about.


Faithful implementation of the existing AMQP specification is clearly a 
requirement as that is central to the charter of the project. Where any 
auxiliary or emerging specifications are adopted by a component, whether 
they are AMQP related or not, they should be compliant with that.


However, having a general policy where all Qpid components are required 
to implement whatever 'emerging standards' come out of the OASIS AMQP 
TCs is not something I am comfortable with.


The Qpid community as a whole needs to have a say in how future features 
will work through an open, collaborative process (open to _anyone_, even 
those primarily involved with other AMQP related projects or products).


Personally I think this would be better for AMQP as well. Allowing and 
encouraging the emergence of grass-roots driven, de-facto 'standards' 
developed in and between open, collaborative and transparent communities 
and backed up by proven interoperable implementations, which can then 
form the basis for official de-jure standardisation.


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid AMQP 1.0 - How does it all hang together? - was Re: Qpid Dispatch Router component

[Rob Godfrey]

This is an excellent post which I think highlights the need for us to
properly define the scopes of our current components, their roadmaps, and
the vision we have for Qpid and AMQP 1.0.

I know some people have already replied in thread, but would it seem like a
good idea if we used these questions to help us formulate some sort of
documentation for our site where we can provide answers both to others
looking to get started with the project, but also perhaps to ourselves?

-- Rob


On 9 October 2013 20:22, Fraser Adams  wrote:

> Hey all,
> The thread below on the dev list has prompted me to ask something that
> I've tentatively mentioned before, but am still a bit embarrassed to raise
> 'cause it probably makes me seem a bit stupid :-( here goes anyway.
>
>
> So I've kind of held off going down the AMQP 1.0 path partly due to lack
> of time, but also partly due to lack of understanding of how it "all hangs
> together", the new website helps a bit - but TBH I'm still left scratching
> my head somewhat.
>
> I'll try to explain:
>
> Now I know that Proton is intended to be a component usable beyond just
> the Qpid "product set", but there's a "protocol engine" and a "messenger
> API" and I'm not even that clear on the relationship between the two of
> those - for example could one use the protocol engine completely
> independently (is there an engine API?) or is the messenger API intended to
> be the lowest "unit of currency", what would be the benefit using the raw
> engine?
>
> Then beyond that there's the relationship with say qpidd and
> qpid::messaging. Now I'm aware that when the Proton libraries are detected
> qpidd and qpid::messaging get built with Proton support, I'm "guessing"
> that in that case the relationship analogous to that of qpid::client where
> qpid::client was the low level AMQP speaking API and qpid::messaging
> provides a higher level abstraction, so I *think* that's the relationship
> with Proton there - but I'm not sure? Is the proton API close to the AMQP
> 1.0 specification in say the way that qpid::client was?
>
>
> But then there's more nuance, so I'm aware that with AMQP 1.0 there's a
> more peer-to-peer relationship and indeed the Proton tests seem to have
> msgr-recv and msgr-send talking directly to each other without a broker. So
> that leads me to ask the question what's the relationship with the broker -
> in other words what services are provided in messenger, what are enhanced
> in qpid::messaging and what are layered on top of that via the broker (and
> how does the addressing and routing work?).
>
> Some examples of where I'm befuddled include how does subscription work at
> a peer to peer level? For example I think that exchange nodes are only
> something I've heard discussed in the context of qpidd and similarly I
> think the same is true of message selectors, so does Proton only provide
> low level network connectivity and data serialisation (and possibly single
> client queue) and all the other stuff needed for connecting a network of
> clients are part of the broker services.
>
> I suppose what I'm really asking is what "services" are provided at each
> "layer" of the Qpid "stack" - clearly you can do useful stuff with just
> Proton - but what stuff and what are the limits? What would you then get
> from qpid:messaging and what then does the broker throw into the mix. Are
> there any diagrams that illustrate this sort of relationship?
>
> The dispatch router adds yet more nuance into the mix. From my (limited)
> understanding it seems to offer at least some of the same services as the
> broker - but I'm not quite sure what. In my case I've got a very large
> federated topology and I have lots of left hand systems feeding in to fewer
> systems towards the right. Given that it's only on the right hand side
> broker that I have lots of consumers doing complex subscriptions and the
> rest of the brokers are employing fairly simple queue routes I'm thinking
> that the dispatch router could ultimately be something to "tidy up" the
> left hand side of my system - but I'm not quite sure.
>
> Apologies if these seem silly questions, I'm sure that the answers are
> obvious to those who've been involved at the architectural stages, but
> ultimately from my perspective the overall holistic architecture isn't
> totally clear.
>
> Even at a basic level I've not actually noticed anything in the
> programming book http://qpid.apache.org/**releases/qpid-0.24/**
> programming/book/index.htmlthat
>  seems to mention even how to connect via AMQP 1.0 vice 0.10. I think
> that it has been mentioned on the mailing list by Gordon so I'm sure I
> could dig the info out, but is it missing from the docs (or am I just not
> looking hard enough). On a similar note for Proton the msgr-send and
> msgr-recv examples are fine as far as it goes, but I'm thinking that to
> figure out how to do anything more complex my best 

Re: Qpid Dispatch Router component

[Rob Godfrey]

On 9 October 2013 19:06, Ted Ross  wrote:

> Rob,
>
> These are good points.  Let me start with management.
>
> I view Dispatch as a bit of a testing ground for the emerging AMQP
> Management specification.  I would claim that at this point, the
> specification is not ready for prime-time.  With regard to the address, the
> specification uses "$management".  Perhaps I'm mistaken, but I took that to
> mean "place-holder for an as-yet unspecified literal string".  Clearly
> there needs to be a way to address multiple distinct container agents
> through the same connection or link.  If this is not the case, then it will
> need to be addressed in the technical committee.
>
>
"$management" is meant as a literal.  When you pair this with global
addressing you can define addresses for each of the management nodes in
separate container instances //$management //$management, etc... moreover you may wish to have the management node at
each instance aware of the management nodes of instances to which it is
connected, thus by issuing the discover commands to the management node at
the local router you have connected to you can potentially find all
management nodes in the connected network.


> I've personally been tracking both the management and addressing
> specifications circulating through the technical committee and I expect
> that Dispatch will conform to (and in fact prove out) both specifications.
>  I'm not aware of any other project within Qpid that is implementing the
> management specification (perhaps the Java broker is).
>
>
My main concern is that I believe that Qpid should be primarily directed at
implementing AMQP standards, and building resuable toolkits and components
that fit into any AMQP network.  I'd be very concerned if we were inventing
alternative management protocols, or building components that only
interoperated with other Qpid tools.

So, personally I'd like to see statement around components saying that they
will be fully implementing emerging AMQP Management, AMQP addressing, etc.
standards, and that we as a project then ensure we stick to these goals.


> With regard to interaction with other Qpid and Apache projects, Dispatch
> is already proved interoperable with the Proton Messenger and
> qpid::messaging clients.  It should also be usable as interconnect between
> clients and the Qpid brokers via AMQP 1.0 and between multiple brokers as a
> federation interconnect.  Some experimentation has been done using the
> ActiveMQ broker as well.
>
> With regard to protocols, I would be opposed to trying to implement AMQP
> 0-* due to the asymmetric nature of those protocols.  I do think, however,
> that Dispatch could make a very good platform for an edge concentrator for
> lighter protocols like MQTT.
>
>
I guess what I'm driving at is that it would be good to have a proper
scope/charter statement for each of our "components" that we as a project
had broad agreement on which do restrict us from just dropping anything we
feel like into a component.  I'd like to do that for Dispatch before we
elevate it, and I think we should ensure that we also have such statements
for the JMS client (which I think should be easy) and Proton (which may be
a little trickier).

Cheers,
Rob


> Regards,
>
> -Ted
>
>
> On 10/09/2013 12:20 PM, Rob Godfrey wrote:
>
>> Hi Ted,
>>
>> I think before we make this a full sub project, it would be good to have
>> clarity on exactly the proposed scope of Dispatch, how it is expected to
>> interact with other components within Qpid, or within wider AMQP networks.
>> I think in retrospect we didn't do this clearly enough with Proton (for
>> example).
>>
>> Moreover I would personally like to understand which AMQP standards it
>> will
>> be looking to implement, and which not.  For instance I notice this line
>> in
>> the docs for Dispatch:
>>
>> *Address**Description* /_local/agentThe management agent on the attached
>>
>> router/container. This address would be used by an endpoint that is a
>> management client/console/tool wishing to access management data from the
>> attached container.
>> Which doesn't seem to conform with the proposed management specification
>> for AMQP, nor does the document make any mention of how dispatch is to be
>> managed.
>>
>>
>> Cheers,
>> Rob
>>
>>
>> On 9 October 2013 17:22, Ted Ross  wrote:
>>
>>  The AMQP Router project (Qpid Dispatch, announced previously on the user
>>> list) is gaining in community interest and is nearing the point where a
>>> first release is appropriate. In preparation for a release, I proposethat
>>> this sub-project follow the lead of both Proton and the AMQP1.0 JMS
>>> projects. This involves:
>>>
>>> 1. Moving the code from qpid/extras to
>>> 
>>> http://svn.apache.org/repos/asf/qpid/dispatch
>>> 
>>> >
>>>
>>> ,
>>> 2. Requesting, by vote, the creation of a JIR

Re: Qpid AMQP 1.0 - How does it all hang together? - was Re: Qpid Dispatch Router component

[Robbie Gemmell]

Adding some more specific bits to what Ted covered in his mail.though I
now see he just beat me to doing so himself, oh well :P

On 9 October 2013 19:22, Fraser Adams  wrote:

> Hey all,
> The thread below on the dev list has prompted me to ask something that
> I've tentatively mentioned before, but am still a bit embarrassed to raise
> 'cause it probably makes me seem a bit stupid :-( here goes anyway.
>
>
> So I've kind of held off going down the AMQP 1.0 path partly due to lack
> of time, but also partly due to lack of understanding of how it "all hangs
> together", the new website helps a bit - but TBH I'm still left scratching
> my head somewhat.
>
> I'll try to explain:
>
> Now I know that Proton is intended to be a component usable beyond just
> the Qpid "product set", but there's a "protocol engine" and a "messenger
> API" and I'm not even that clear on the relationship between the two of
> those - for example could one use the protocol engine completely
> independently (is there an engine API?) or is the messenger API intended to
> be the lowest "unit of currency", what would be the benefit using the raw
> engine?
>

The engine is effectively an AMQP 1.0 implementation that can be used to
add support to a [potentially existing] product without requiring
implementation of AMQP 1.0 from scrach, e.g. this is what ActiveMQ and Qpid
C++ brokers do for their 1.0 support. Messenger is a form of API for
applications doing messaging that uses the underlying engine as its AMQP
1.0 implementation, and I'll now defer to others to better describe it
beyond that :)


>
> Then beyond that there's the relationship with say qpidd and
> qpid::messaging. Now I'm aware that when the Proton libraries are detected
> qpidd and qpid::messaging get built with Proton support, I'm "guessing"
> that in that case the relationship analogous to that of qpid::client where
> qpid::client was the low level AMQP speaking API and qpid::messaging
> provides a higher level abstraction, so I *think* that's the relationship
> with Proton there - but I'm not sure? Is the proton API close to the AMQP
> 1.0 specification in say the way that qpid::client was?
>

Similar, but not entirely the same, as Qpid::client was the actual C++
client interface for some time but as mentioned was close to the
specification itself, and so a higher level messaging API was later created
to abstract away some of that detail and provide a replacement messaging
API that could allow easier transition to newer protocols. Proton is in
part something that could be used to underneath API X to make it support
AMQP 1.0 (e.g. the qpid messaging API), but there also isnt necessarily a
client API above it at and it could jsut be getting used directly (e.g
possibly with the brokers).


>
>
> But then there's more nuance, so I'm aware that with AMQP 1.0 there's a
> more peer-to-peer relationship and indeed the Proton tests seem to have
> msgr-recv and msgr-send talking directly to each other without a broker. So
> that leads me to ask the question what's the relationship with the broker -
> in other words what services are provided in messenger, what are enhanced
> in qpid::messaging and what are layered on top of that via the broker (and
> how does the addressing and routing work?).
>

You can stick a broker in the middle and the above just becomes a
client<->broker<->client model like you are used to. Depending on what you
are actually doing though, you might not need or want a broker, and that
field is likely to develop as time goes. For example, brokers can do things
like provide Queues, but that is essentially just something with a name
that accepts your messages...you migiht want to be sending to something
with a name that isnt a queue on a broker.


>
> Some examples of where I'm befuddled include how does subscription work at
> a peer to peer level? For example I think that exchange nodes are only
> something I've heard discussed in the context of qpidd and similarly I
> think the same is true of message selectors, so does Proton only provide
> low level network connectivity and data serialisation (and possibly single
> client queue) and all the other stuff needed for connecting a network of
> clients are part of the broker services.
>
> I suppose what I'm really asking is what "services" are provided at each
> "layer" of the Qpid "stack" - clearly you can do useful stuff with just
> Proton - but what stuff and what are the limits? What would you then get
> from qpid:messaging and what then does the broker throw into the mix. Are
> there any diagrams that illustrate this sort of relationship?
>

I think all I would say here would be to echo my points above: you can end
up with something that very closely resembles what you are used to, i.e.
clients talking directly to brokers and using queues and topics, or
depending on your needs you might want to end up with something different.


>
> The dispatch router adds yet more nuance into the mix. From my (limite

Re: Qpid AMQP 1.0 - How does it all hang together? - was Re: Qpid Dispatch Router component

[Ted Ross]

Now, to answer your actual questions...  See inline below.

On 10/09/2013 02:22 PM, Fraser Adams wrote:

Hey all,
The thread below on the dev list has prompted me to ask something that 
I've tentatively mentioned before, but am still a bit embarrassed to 
raise 'cause it probably makes me seem a bit stupid :-( here goes 
anyway.



So I've kind of held off going down the AMQP 1.0 path partly due to 
lack of time, but also partly due to lack of understanding of how it 
"all hangs together", the new website helps a bit - but TBH I'm still 
left scratching my head somewhat.


I'll try to explain:

Now I know that Proton is intended to be a component usable beyond 
just the Qpid "product set", but there's a "protocol engine" and a 
"messenger API" and I'm not even that clear on the relationship 
between the two of those - for example could one use the protocol 
engine completely independently (is there an engine API?) or is the 
messenger API intended to be the lowest "unit of currency", what would 
be the benefit using the raw engine?


Proton is a multi-layered thing.  There is an "engine" API, but it's 
neither documented nor stabilized.  The Messenger API is built on top of 
the engine API and is better documented.  There's also a separate driver 
component in Proton for integration into a particular run-time environment.


The benefit of using the raw engine API is that it gives detailed access 
to the specific parts of the AMQP protocol.  If you need to work 
directly with connections, sessions, links and deliveries, the engine 
API is what you need.


Dispatch is built on the engine API (the C language one) and bypasses 
all the services offered by Messenger.  There's a layer within Dispatch 
that provides multi-threaded access to engine (which is by itself, 
non-thread-safe).




Then beyond that there's the relationship with say qpidd and 
qpid::messaging. Now I'm aware that when the Proton libraries are 
detected qpidd and qpid::messaging get built with Proton support, I'm 
"guessing" that in that case the relationship analogous to that of 
qpid::client where qpid::client was the low level AMQP speaking API 
and qpid::messaging provides a higher level abstraction, so I *think* 
that's the relationship with Proton there - but I'm not sure? Is the 
proton API close to the AMQP 1.0 specification in say the way that 
qpid::client was?


This is sort of true.  The qpid::client API was the original one and it 
was tied pretty tightly to the protocol.  The qpid::messaging APIs (both 
C++ and Python) were created prior to AMQP 1.0 for the purpose of being 
forward compatible to the new protocol.  It is simply an implementation 
detail that qpid::messaging is layered on qpid::client.


Messenger and qpid::messaging are peers.  They are alternative APIs for 
messaging.  Messenger is more abstract in that it deals with messages 
and hides connections, sessions, links, etc. Qpid::messaging is closer 
to JMS in its design.





But then there's more nuance, so I'm aware that with AMQP 1.0 there's 
a more peer-to-peer relationship and indeed the Proton tests seem to 
have msgr-recv and msgr-send talking directly to each other without a 
broker. So that leads me to ask the question what's the relationship 
with the broker - in other words what services are provided in 
messenger, what are enhanced in qpid::messaging and what are layered 
on top of that via the broker (and how does the addressing and routing 
work?).


This is a pretty big question...  AMQP 1.0 is a point-to-point, 
symmetric protocol.  Because of this, there is no requirement for a 
broker.  It allows for the full semantics of message exchange to occur 
between endpoints regardless of whether one (or both) of them is a broker.


Regarding addressing and routing.  This is a topic of research. Dispatch 
is one research vehicle for these topics.  It takes a particular 
approach to addressing and routing that may be different from other 
approaches.  We should explore them all.




Some examples of where I'm befuddled include how does subscription 
work at a peer to peer level? For example I think that exchange nodes 
are only something I've heard discussed in the context of qpidd and 
similarly I think the same is true of message selectors, so does 
Proton only provide low level network connectivity and data 
serialisation (and possibly single client queue) and all the other 
stuff needed for connecting a network of clients are part of the 
broker services.


Pub-sub in a brokerless environment is another topic of research. 
Dispatch is attempting to solve this problem.  I expect others will as well.




I suppose what I'm really asking is what "services" are provided at 
each "layer" of the Qpid "stack" - clearly you can do useful stuff 
with just Proton - but what stuff and what are the limits? What would 
you then get from qpid:messaging and what then does the broker throw 
into the mix. Are there any diagrams that illustrate this sort of 
rel

Re: Qpid AMQP 1.0 - How does it all hang together? - was Re: Qpid Dispatch Router component

[William Henry]

Ted, 

Can someone add your text to the community web pages and link parts to 
appropriate pages?

i.e. can we use Frase's and your posts to augment the community pages rather 
than just leaving here in list archives?

William 

- Original Message -
> Frase,
> 
> This is an excellent post, and I believe quite relevant.  I'll try to
> address your questions at an abstract level rather than point-by-point.
> Your confusion is not unique, but quite justified.
> 
> AMQP 1.0 is simply a wire-level protocol specification for symmetric
> point-to-point data communication.  It has very advanced semantic
> capabilities that are inspired by middleware messaging.  AMQP is, in my
> opinion, a game changer for distributed computing.  It opens up
> possibilities that go way beyond client/broker messaging.  In fact, it
> holds the potential to revolutionize the much broader world of data
> networking and distributed systems because it offers capabilities that
> TCP/IP and HTTP can't touch.
> 
> Apache Qpid is the premier open source implementation of AMQP.  In the
> AMQP 0-{8,9,10} era, Qpid was pretty much just another
> middleware-oriented-messaging (MOM) system based on the emerging
> standard.  With the ratification of AMQP 1.0, which is significantly
> different (and superior) to the predecessor specifications, Qpid is
> undergoing a bit of a transition.
> 
> We have, of course, updated our brokers and clients to use the new
> specification and thus continue to serve the MOM use cases.
> 
> The Proton project was created for the purpose of promoting AMQP and
> making it easy to integrate into many different environments (I believe
> we've lost sight of that purpose somewhat).
> 
> The Dispatch project was created for the purpose of exploring how AMQP
> can be applied in non-broker networks.  Dispatch is a form of advanced
> interconnect based on AMQP with the goal of building large-scale and
> high-performance networks that take advantage of the unique capabilities
> of the protocol.  It is intended to extend the reach and usefulness of
> brokers.
> 
> There are a lot of new ideas floating around, some of them overlapping.
> I think Qpid is a perfect place to explore and develop new technologies
> based on AMQP.  This will cause some confusion and force us to work at
> articulating what we are doing and thinking, and it will be people like
> you who will prompt the important discussions.
> 
> Regards,
> 
> -Ted
> 
> 
> On 10/09/2013 02:22 PM, Fraser Adams wrote:
> > Hey all,
> > The thread below on the dev list has prompted me to ask something that
> > I've tentatively mentioned before, but am still a bit embarrassed to
> > raise 'cause it probably makes me seem a bit stupid :-( here goes
> > anyway.
> >
> >
> > So I've kind of held off going down the AMQP 1.0 path partly due to
> > lack of time, but also partly due to lack of understanding of how it
> > "all hangs together", the new website helps a bit - but TBH I'm still
> > left scratching my head somewhat.
> >
> > I'll try to explain:
> >
> > Now I know that Proton is intended to be a component usable beyond
> > just the Qpid "product set", but there's a "protocol engine" and a
> > "messenger API" and I'm not even that clear on the relationship
> > between the two of those - for example could one use the protocol
> > engine completely independently (is there an engine API?) or is the
> > messenger API intended to be the lowest "unit of currency", what would
> > be the benefit using the raw engine?
> >
> > Then beyond that there's the relationship with say qpidd and
> > qpid::messaging. Now I'm aware that when the Proton libraries are
> > detected qpidd and qpid::messaging get built with Proton support, I'm
> > "guessing" that in that case the relationship analogous to that of
> > qpid::client where qpid::client was the low level AMQP speaking API
> > and qpid::messaging provides a higher level abstraction, so I *think*
> > that's the relationship with Proton there - but I'm not sure? Is the
> > proton API close to the AMQP 1.0 specification in say the way that
> > qpid::client was?
> >
> >
> > But then there's more nuance, so I'm aware that with AMQP 1.0 there's
> > a more peer-to-peer relationship and indeed the Proton tests seem to
> > have msgr-recv and msgr-send talking directly to each other without a
> > broker. So that leads me to ask the question what's the relationship
> > with the broker - in other words what services are provided in
> > messenger, what are enhanced in qpid::messaging and what are layered
> > on top of that via the broker (and how does the addressing and routing
> > work?).
> >
> > Some examples of where I'm befuddled include how does subscription
> > work at a peer to peer level? For example I think that exchange nodes
> > are only something I've heard discussed in the context of qpidd and
> > similarly I think the same is true of message selectors, so does
> > Proton only provide low level network connectivity and data
> 

Re: Qpid AMQP 1.0 - How does it all hang together? - was Re: Qpid Dispatch Router component

[William Henry]

Great post! +1.

The community needs to do more to explain, document, and promote.

William

- Original Message -
> Hey all,
> The thread below on the dev list has prompted me to ask something that
> I've tentatively mentioned before, but am still a bit embarrassed to
> raise 'cause it probably makes me seem a bit stupid :-( here goes
> anyway.
> 
> 
> So I've kind of held off going down the AMQP 1.0 path partly due to lack
> of time, but also partly due to lack of understanding of how it "all
> hangs together", the new website helps a bit - but TBH I'm still left
> scratching my head somewhat.
> 
> I'll try to explain:
> 
> Now I know that Proton is intended to be a component usable beyond just
> the Qpid "product set", but there's a "protocol engine" and a "messenger
> API" and I'm not even that clear on the relationship between the two of
> those - for example could one use the protocol engine completely
> independently (is there an engine API?) or is the messenger API intended
> to be the lowest "unit of currency", what would be the benefit using the
> raw engine?
> 
> Then beyond that there's the relationship with say qpidd and
> qpid::messaging. Now I'm aware that when the Proton libraries are
> detected qpidd and qpid::messaging get built with Proton support, I'm
> "guessing" that in that case the relationship analogous to that of
> qpid::client where qpid::client was the low level AMQP speaking API and
> qpid::messaging provides a higher level abstraction, so I *think* that's
> the relationship with Proton there - but I'm not sure? Is the proton API
> close to the AMQP 1.0 specification in say the way that qpid::client was?
> 
> 
> But then there's more nuance, so I'm aware that with AMQP 1.0 there's a
> more peer-to-peer relationship and indeed the Proton tests seem to have
> msgr-recv and msgr-send talking directly to each other without a broker.
> So that leads me to ask the question what's the relationship with the
> broker - in other words what services are provided in messenger, what
> are enhanced in qpid::messaging and what are layered on top of that via
> the broker (and how does the addressing and routing work?).
> 
> Some examples of where I'm befuddled include how does subscription work
> at a peer to peer level? For example I think that exchange nodes are
> only something I've heard discussed in the context of qpidd and
> similarly I think the same is true of message selectors, so does Proton
> only provide low level network connectivity and data serialisation (and
> possibly single client queue) and all the other stuff needed for
> connecting a network of clients are part of the broker services.
> 
> I suppose what I'm really asking is what "services" are provided at each
> "layer" of the Qpid "stack" - clearly you can do useful stuff with just
> Proton - but what stuff and what are the limits? What would you then get
> from qpid:messaging and what then does the broker throw into the mix.
> Are there any diagrams that illustrate this sort of relationship?
> 
> The dispatch router adds yet more nuance into the mix. From my (limited)
> understanding it seems to offer at least some of the same services as
> the broker - but I'm not quite sure what. In my case I've got a very
> large federated topology and I have lots of left hand systems feeding in
> to fewer systems towards the right. Given that it's only on the right
> hand side broker that I have lots of consumers doing complex
> subscriptions and the rest of the brokers are employing fairly simple
> queue routes I'm thinking that the dispatch router could ultimately be
> something to "tidy up" the left hand side of my system - but I'm not
> quite sure.
> 
> Apologies if these seem silly questions, I'm sure that the answers are
> obvious to those who've been involved at the architectural stages, but
> ultimately from my perspective the overall holistic architecture isn't
> totally clear.
> 
> Even at a basic level I've not actually noticed anything in the
> programming book
> http://qpid.apache.org/releases/qpid-0.24/programming/book/index.html
> that seems to mention even how to connect via AMQP 1.0 vice 0.10. I
> think that it has been mentioned on the mailing list by Gordon so I'm
> sure I could dig the info out, but is it missing from the docs (or am I
> just not looking hard enough). On a similar note for Proton the
> msgr-send and msgr-recv examples are fine as far as it goes, but I'm
> thinking that to figure out how to do anything more complex my best bet
> is likely to be to "reverse engineer" the qpid::messaging bindings - I
> can't see anything obvious for how to send a map for example. I'm
> guessing that Proton is just as erm "nuanced" as qpid::client, so really
> powerful and flexible, but you have to know what you're doing to get the
> best (say performance) out of it, the API documentation looks pretty
> decent to be fair but I'm not sure that's enough to help me drive it
> really effectively.
> 
> On top o

Re: Qpid AMQP 1.0 - How does it all hang together? - was Re: Qpid Dispatch Router component

[Ted Ross]

Frase,

This is an excellent post, and I believe quite relevant.  I'll try to 
address your questions at an abstract level rather than point-by-point.  
Your confusion is not unique, but quite justified.


AMQP 1.0 is simply a wire-level protocol specification for symmetric 
point-to-point data communication.  It has very advanced semantic 
capabilities that are inspired by middleware messaging.  AMQP is, in my 
opinion, a game changer for distributed computing.  It opens up 
possibilities that go way beyond client/broker messaging.  In fact, it 
holds the potential to revolutionize the much broader world of data 
networking and distributed systems because it offers capabilities that 
TCP/IP and HTTP can't touch.


Apache Qpid is the premier open source implementation of AMQP.  In the 
AMQP 0-{8,9,10} era, Qpid was pretty much just another 
middleware-oriented-messaging (MOM) system based on the emerging 
standard.  With the ratification of AMQP 1.0, which is significantly 
different (and superior) to the predecessor specifications, Qpid is 
undergoing a bit of a transition.


We have, of course, updated our brokers and clients to use the new 
specification and thus continue to serve the MOM use cases.


The Proton project was created for the purpose of promoting AMQP and 
making it easy to integrate into many different environments (I believe 
we've lost sight of that purpose somewhat).


The Dispatch project was created for the purpose of exploring how AMQP 
can be applied in non-broker networks.  Dispatch is a form of advanced 
interconnect based on AMQP with the goal of building large-scale and 
high-performance networks that take advantage of the unique capabilities 
of the protocol.  It is intended to extend the reach and usefulness of 
brokers.


There are a lot of new ideas floating around, some of them overlapping.  
I think Qpid is a perfect place to explore and develop new technologies 
based on AMQP.  This will cause some confusion and force us to work at 
articulating what we are doing and thinking, and it will be people like 
you who will prompt the important discussions.


Regards,

-Ted


On 10/09/2013 02:22 PM, Fraser Adams wrote:

Hey all,
The thread below on the dev list has prompted me to ask something that 
I've tentatively mentioned before, but am still a bit embarrassed to 
raise 'cause it probably makes me seem a bit stupid :-( here goes 
anyway.



So I've kind of held off going down the AMQP 1.0 path partly due to 
lack of time, but also partly due to lack of understanding of how it 
"all hangs together", the new website helps a bit - but TBH I'm still 
left scratching my head somewhat.


I'll try to explain:

Now I know that Proton is intended to be a component usable beyond 
just the Qpid "product set", but there's a "protocol engine" and a 
"messenger API" and I'm not even that clear on the relationship 
between the two of those - for example could one use the protocol 
engine completely independently (is there an engine API?) or is the 
messenger API intended to be the lowest "unit of currency", what would 
be the benefit using the raw engine?


Then beyond that there's the relationship with say qpidd and 
qpid::messaging. Now I'm aware that when the Proton libraries are 
detected qpidd and qpid::messaging get built with Proton support, I'm 
"guessing" that in that case the relationship analogous to that of 
qpid::client where qpid::client was the low level AMQP speaking API 
and qpid::messaging provides a higher level abstraction, so I *think* 
that's the relationship with Proton there - but I'm not sure? Is the 
proton API close to the AMQP 1.0 specification in say the way that 
qpid::client was?



But then there's more nuance, so I'm aware that with AMQP 1.0 there's 
a more peer-to-peer relationship and indeed the Proton tests seem to 
have msgr-recv and msgr-send talking directly to each other without a 
broker. So that leads me to ask the question what's the relationship 
with the broker - in other words what services are provided in 
messenger, what are enhanced in qpid::messaging and what are layered 
on top of that via the broker (and how does the addressing and routing 
work?).


Some examples of where I'm befuddled include how does subscription 
work at a peer to peer level? For example I think that exchange nodes 
are only something I've heard discussed in the context of qpidd and 
similarly I think the same is true of message selectors, so does 
Proton only provide low level network connectivity and data 
serialisation (and possibly single client queue) and all the other 
stuff needed for connecting a network of clients are part of the 
broker services.


I suppose what I'm really asking is what "services" are provided at 
each "layer" of the Qpid "stack" - clearly you can do useful stuff 
with just Proton - but what stuff and what are the limits? What would 
you then get from qpid:messaging and what then does the broker throw 
into the mix. Are there any dia

Qpid AMQP 1.0 - How does it all hang together? - was Re: Qpid Dispatch Router component

[Fraser Adams]

Hey all,
The thread below on the dev list has prompted me to ask something that 
I've tentatively mentioned before, but am still a bit embarrassed to 
raise 'cause it probably makes me seem a bit stupid :-( here goes 
anyway.



So I've kind of held off going down the AMQP 1.0 path partly due to lack 
of time, but also partly due to lack of understanding of how it "all 
hangs together", the new website helps a bit - but TBH I'm still left 
scratching my head somewhat.


I'll try to explain:

Now I know that Proton is intended to be a component usable beyond just 
the Qpid "product set", but there's a "protocol engine" and a "messenger 
API" and I'm not even that clear on the relationship between the two of 
those - for example could one use the protocol engine completely 
independently (is there an engine API?) or is the messenger API intended 
to be the lowest "unit of currency", what would be the benefit using the 
raw engine?


Then beyond that there's the relationship with say qpidd and 
qpid::messaging. Now I'm aware that when the Proton libraries are 
detected qpidd and qpid::messaging get built with Proton support, I'm 
"guessing" that in that case the relationship analogous to that of 
qpid::client where qpid::client was the low level AMQP speaking API and 
qpid::messaging provides a higher level abstraction, so I *think* that's 
the relationship with Proton there - but I'm not sure? Is the proton API 
close to the AMQP 1.0 specification in say the way that qpid::client was?



But then there's more nuance, so I'm aware that with AMQP 1.0 there's a 
more peer-to-peer relationship and indeed the Proton tests seem to have 
msgr-recv and msgr-send talking directly to each other without a broker. 
So that leads me to ask the question what's the relationship with the 
broker - in other words what services are provided in messenger, what 
are enhanced in qpid::messaging and what are layered on top of that via 
the broker (and how does the addressing and routing work?).


Some examples of where I'm befuddled include how does subscription work 
at a peer to peer level? For example I think that exchange nodes are 
only something I've heard discussed in the context of qpidd and 
similarly I think the same is true of message selectors, so does Proton 
only provide low level network connectivity and data serialisation (and 
possibly single client queue) and all the other stuff needed for 
connecting a network of clients are part of the broker services.


I suppose what I'm really asking is what "services" are provided at each 
"layer" of the Qpid "stack" - clearly you can do useful stuff with just 
Proton - but what stuff and what are the limits? What would you then get 
from qpid:messaging and what then does the broker throw into the mix. 
Are there any diagrams that illustrate this sort of relationship?


The dispatch router adds yet more nuance into the mix. From my (limited) 
understanding it seems to offer at least some of the same services as 
the broker - but I'm not quite sure what. In my case I've got a very 
large federated topology and I have lots of left hand systems feeding in 
to fewer systems towards the right. Given that it's only on the right 
hand side broker that I have lots of consumers doing complex 
subscriptions and the rest of the brokers are employing fairly simple 
queue routes I'm thinking that the dispatch router could ultimately be 
something to "tidy up" the left hand side of my system - but I'm not 
quite sure.


Apologies if these seem silly questions, I'm sure that the answers are 
obvious to those who've been involved at the architectural stages, but 
ultimately from my perspective the overall holistic architecture isn't 
totally clear.


Even at a basic level I've not actually noticed anything in the 
programming book 
http://qpid.apache.org/releases/qpid-0.24/programming/book/index.html 
that seems to mention even how to connect via AMQP 1.0 vice 0.10. I 
think that it has been mentioned on the mailing list by Gordon so I'm 
sure I could dig the info out, but is it missing from the docs (or am I 
just not looking hard enough). On a similar note for Proton the 
msgr-send and msgr-recv examples are fine as far as it goes, but I'm 
thinking that to figure out how to do anything more complex my best bet 
is likely to be to "reverse engineer" the qpid::messaging bindings - I 
can't see anything obvious for how to send a map for example. I'm 
guessing that Proton is just as erm "nuanced" as qpid::client, so really 
powerful and flexible, but you have to know what you're doing to get the 
best (say performance) out of it, the API documentation looks pretty 
decent to be fair but I'm not sure that's enough to help me drive it 
really effectively.


On top of that there seems to be a growing number of JMS clients, 
there's the original AMQP 0.10, there's an AMQP 1.0 one in the main Qpid 
tree and there's a separate Proton based AMQP 1.0 one that's a separate 
component (in a sim