Hi Christopher, this is timely, I started peeking at federation this week also, to see if I can make it a "better bridge" from the perspective of only moving messages that are needed. The idea is to use AMQP as the protocol and flow messages across the bridge based on aggregate AMQP credit, ie: rather than have all messages move between brokers when local consumers are slow, only move to satisfy remote/upstream credit and react to it dynamically, which is a fundamental part of AMQP flow control.
i need to pull together a POC of this to verify how easy/hard it will be to aggregate credit demand etc and have outbound AMQP calls, but I think it can be really good and fix an age old problem with the 5.x bridge. it would also help with the duplex part because of the symmetric nature of AMQP. on the duplex and configuration command, authentication was one problem in 5.x, in that the same users needed to exist on all brokers b/c the user/pass etc was part of the bridge config, I think the "reuse of the same connection" may be important to avoid that need. It will typically need to be TLS and maybe cert based authentication so maybe SASL would also come into play. The duplex case in my mind was always about hub/spoke where the hub did not need to be aware of the spokes configuration. Each spoke could initiate a duplex/two way bridge to the hub and not require any additional fire wall ports. To my mind, propagation of config and reuse of the connection was always related. But for sure small steps. And maybe AMQP can help! On Thu, 17 Oct 2019 at 16:34, Christopher Shannon <christopher.l.shan...@gmail.com> wrote: > > Duplex is still up in the air as I was going to do the downstream portion > first. A true duplex bridge would share the same connection which is what > happens In 5.x. It establishes the bridge and then the remote broker gets > a command to also send messages back over the same connection. > > So we could do something similar, or we could make it easier and just > automatically create two connections. So for example we could define a > duplex connection as part of the federation config and under the covers the > federation will just create 1 upstream and 1 downstream connection > automatically. Having 2 connections could be better performance anyways > and prevent traffic from each direction from getting in the way of the > other. We could also support both options, etc. > > On Thu, Oct 17, 2019 at 11:26 AM Justin Bertram <jbert...@apache.org> wrote: > > > I think your implementation idea makes sense and it is actually quite > > similar to what is done for clustering (i.e. each broker tells all the > > other brokers how they can connect back to it). This makes sense to me as a > > way to configure downstream brokers, but I'm still fuzzy on the "duplex" > > part. Does this idea fulfill both the configuration aspect and the "duplex" > > aspect? Could you clarify what you mean by "duplex"? I always conceived > > that implementing "duplex" would require modifying the bridge to be able to > > "pull" messages rather than only "push" them. > > > > > > Justin > > > > On Thu, Oct 17, 2019 at 8:13 AM Christopher Shannon < > > christopher.l.shan...@gmail.com> wrote: > > > > > I recently started to dive into the federation support as I try and > > migrate > > > 5.x brokers to Artemis as I need something similar to how 5.x does > > bridging > > > and federated queues/addresses seem like more in line to what I need than > > > clustering. > > > > > > However, I've noticed several shortcomings and enhancements that will be > > > necessary to make it useful. The first thing is right now you can only > > > configure an upstream broker which is backwards from how 5.x configures a > > > bridge (it configures a one way downstream). So I wanted to go ahead and > > > enhance Federation support to allow configuring both downstream brokers > > and > > > hopefully duplex as well. > > > > > > For the approach I was thinking that maybe if we could add a > > configuration > > > option for downstream brokers. Then, when the connection is made to the > > > remote broker we could send a new CORE packet command with the info for > > the > > > Federation config. Then the remote broker could receive this config, > > parse > > > it, and then establish an upstream link based on that information back to > > > the broker that made the connection...essentially creating a downstream > > > link but re-using the existing upstream way of creating the bridge to > > > simplify things. > > > > > > I can work on the PR and difference enhancements but wanted to get some > > > agreement on the approach before spending a bunch of time on it. > > > > > > Thoughts? Or other ideas on how to accomplish configuring a downstream > > > broker? > > > > >
