Agreed with all Ted says below about waypoints. I think we currently
lack a clear overview of all the dispatch concepts - the implementation
is on track, we just need to improve the explanation. Here's a quick
thinking-out-loud attempt to summarize:
----
The key to dispatch's flexibility is the "address". For clients that
produce or consume messages, an address is (mostly) an opaque string.
The power of dispatch is that the meaning of addresses can be
statically or dynamically changed in the dispatch network configuration
without updating large numbers of clients. So it is important to
explain exactly what an "address" is.
First problem: there are two sometimes separate, sometimes overlapping
parts to an "address". First you need a network connection to a
dispatch router, so you need a "connection address" typically an IP
host and port along with SSL or SASL configuration. This part of the
address can be managed with standard IP tricks: from static
configuration to DNS balancing, dynamic IP etc.
Next, dispatch operates on a "routable address". The line between these
is blurred. Often we use a URL-style `amqp://host:port/path` that
contains both. To add to the confusion, it is application-dependent
whether we use only `path` as the routable address, or whether we
include some or all of the `host:port`. Dispatch works either way but
we need some clarification and guidance for users.
For the remainder of this discussion "address" means the dispatch
"routable address". There are several address behaviors that can be
configured.
*Special* addresses:
- addresses starting with `_local/` and `_topo/` are reserved special
dispatch use.
- `$management` is reserved for AMQP standard management
[Question should we follow the AMQP convention and use $local/$topo?]
*Mobile* addresses: these are arbitrary application-defined strings.
There have policy associated with them (e.g. mutlicast, load-balanced
etc.) but are not attached to any point in the network. Any client can
send or receive from anywhere the network dynamically routes according
to policy.
*Link-Routed Addresses*: These addresses are associated with an
external AMQP system such as a broker. You can send or receive from
anywhere in the network, but doing so establishes an AMQP "link" which
acts like a "tunnel" through the network directly to the external
system. No dispatch-level policies apply - dispatch simply acts as a go
-between, the external system decides what to do with the messages.
*Waypoint Addresses*: These addresses are also associated with an
external system or systems but in a more loosely coupled way. Waypoints
can have policy associated and be connected to more than one external
AMQP system on the same logical address.
Link-routes vs. waypoints: link routes are best when you want flexible
but direct connections to existing AMQP systems, for example to connect
directly to a broker without knowing exactly where it is hosted.
Waypoints allow you to create more flexible orchestrations of AMQP
systems. For example you can distribute the load to/from a single
logical address over queues or topics on multiple brokers, without the
clients or the brokers being aware of it.
I have some thoughts about making the 3 address types more consistent,
will send another email
On Mon, 2015-11-16 at 08:29 -0500, Ted Ross wrote:
> Hi Noel,
>
> Phased addresses are a Dispatch Router concept, not an AMQP concept.
>
> To illustrate, think of a queue in a broker called "orders". From
> the
> AMQP standpoint, the address of the queue is "orders", but from the
> router-network standpoint, there are actually two distinct addresses
> that need to be independently routed. Messages produced _to_
> "orders"
> can originate anywhere in the network and must be routed to the
> broker.
> Messages _from_ "orders" originate from the broker and must be
> routed
> to consumers which can be anywhere in the network. So we use the
> convention that produced messages are phase-0 and consumed messages
> are
> phase-1.
>
> To get technical, phases are an annotation on an address that allow
> the
> router fast-path in Dispatch to treat each phase of an address as a
> completely distinct address, with its own routing destinations and
> semantics.
>
> In the above example, a waypoint is provisioned to represent and
> manage
> message flow to and from the queue. Each waypoint has an address and
> (typically) two phases, one for deliveries flowing to the waypoint
> and
> another for deliveries flowing from the waypoint.
>
> It's important to note that from the broker's perspective and also
> that
> of the clients that produce and consume messages, this is all
> transparent. Clients and brokers only use the base address
> ("orders")
> and are unaware of the phased message routing.
>
> So here are a few implications:
>
> 1) You can have multiple waypoints on the same address and with the
> same
> phases (i.e. multiple brokers with queues named "orders" using phases
> 0
> and 1). Dispatch can be configured to anycast (balance) messages to
> and
> from the waypoints, thus balancing the order load across multiple
> brokers in a way that is transparent to clients. Furthermore, the
> number of waypoints can be changed dynamically without the knowledge
> of
> the clients (or brokers) to react to changes in order load. (Note
> that
> the ability to quiesce and remove waypoints is not yet implemented).
>
> 2) Waypoints can be arranged serially (i.e. a 0->1 waypoint can feed
> a
> 1->2 waypoint, etc.). Waypoints can be broker queues or topics.
> They
> can also be arbitrary processes that receive and send messages,
> perhaps
> doing some form of transformation, filtering, or coalescing of data.
>
> 3) Waypoints can be single-phase. You can have a "source" waypoint
> that
> only has a "from" phase. Messages come out but don't go in.
> Likewise,
> a "sink" waypoint can be created that only receives messages.
>
> 4) Different phases of an address can have different semantics (i.e.
> multicast vs. anycast). You can set up a waypoint such that produced
> messages are balanced (anycast) across instances of the waypoint and
> that messages from the waypoint are broadcast to all attached
> consumers.
>
> I hope this helps. Waypoints are still in active development, so
> your
> feedback and ideas are most welcome.
>
> -Ted
>
>
> On 11/15/2015 05:51 AM, Noel OConnor wrote:
> > Hi,
> > I've struggling with understanding the context of phases in
> > waypoints and
> > how you would use them.
> > Are phases an AMQP or dispatch routes concept ?
> >
> > I've taken a look at the code and found the following
> >
> > >>* A waypoint sends/receives messages to/from an external entity
> > such as a
> > >>* broker as part of a multi-phase address.
> >
> > Is a multi-phase address something line a/b/c/d ?
> >
> > >>* An address can have multiple phases. Each phase acts like a
> > separate
> > address,
> > >>* but sharing the same address string.
> >
> > >>* Phases are not visible to normal senders/receivers, they are
> > set by
> > >>* waypoints. Messages from normal senders go to the phase=0
> > address.
> > Normal
> > >>* subscribers subscribe to the highest phase defined for the
> > address.
> >
> > >>* A waypoint takes messages for its in-phase and sends them to
> > the
> > external
> > >>* entity. Messages received from the external entity are given
> > the
> > waypoint's
> > >>* out-phase. Waypoints can be "chained" with the out-phase of
> > one equal
> > to the
> > >>* in-phase for the next. Thus waypoints provide a way to route
> > messages
> > via
> > >>* multiple external entities between a sender and a subscriber
> > using the
> > same
> > >>* address.
> >
> > An example of the above would be great ?
> > Are waypoints a kind of orchestration scheme i.e. routing from one
> > broker
> > to another ?
> >
> > thanks for your help
> > Noel
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
