Typically we like to keep everything on the list so the whole community can benefit from the discussion. However, if there's a specific reason that privacy is a concern then you can email me directly or you can find me on the ASF Slack in #activemq.
Justin On Tue, Mar 28, 2023 at 7:56 AM Roy Cohen <roy_co...@hotmail.com> wrote: > Hi Justin > > Would it be able to reach out to you directly to discuss a couple of > points around the discussion below privately ? > > Thanks > Roy > > > > On 28 Mar 2023, at 10:26, Roy Cohen <roy_co...@hotmail.com> wrote: > > > > You have indeed ! :) > > > > On 27 Mar 2023, at 19:35, Justin Bertram <jbert...@apache.org> wrote: > > > > I wrote that on a completely different thread [1] related to MQTT > retained > > messages in a cluster. It is not related to this thread or your issue > > generally. > > > > > > Justin > > > > [1] https://lists.apache.org/thread/oq41shfpv108m739km3rhs4tfj76c1zf > > > > On Mon, Mar 27, 2023 at 1:28 PM Roy Cohen <roy_co...@hotmail.com> wrote: > > > >> To quote: > >> > >> “This functionality isn't supported, and while it may be technically > >> feasible to implement I'm not sure how much sense it makes overall.” > >> > >> On 27 Mar 2023, at 19:16, Justin Bertram <jbert...@apache.org> wrote: > >> > >> I'm not sure where I may have indicated that either one of those things > >> isn't supported. > >> > >> In any case, you can do either. > >> > >> > >> Justin > >> > >> On Mon, Mar 27, 2023 at 1:07 PM Roy Cohen <roy_co...@hotmail.com> > wrote: > >> > >>> Just to be clear: When you say “isn’t supported” do you mean a third > >>> broker or co located backups when running each broker on its own VM ? > >>> > >>>> On 27 Mar 2023, at 19:04, Roy Cohen <roy_co...@hotmail.com> wrote: > >>>> > >>>> Will do Justin and many thanks for all the additional details which I > >>> will certainly bring forward internally, much appreciated > >>>> > >>>> On 27 Mar 2023, at 18:58, Justin Bertram <jbert...@apache.org> wrote: > >>>> > >>>> I recently added a new section to the clustering documentation > >> regarding > >>>> things to keep in mind regarding performance [1]. > >>>> > >>>> Also, it's worth noting that often the bottleneck in messaging is not > >> the > >>>> broker itself but rather the consumer(s). It might be worth ensuring > >> that > >>>> the bottleneck really is the broker. As noted in the new documentation > >>> [1], > >>>> adding brokers to a cluster can actually *reduce* throughput in > certain > >>>> circumstances. > >>>> > >>>> Let me know if using group-name works for you. > >>>> > >>>> > >>>> Justin > >>>> > >>>> [1] > >>>> > >>> > >> > https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations > >>>> > >>>> On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <roy_co...@hotmail.com> > >>> wrote: > >>>> > >>>>> I haven’t tried he group-name yet. > >>>>> > >>>>> With regards to the third broker: The architects believe it’ll > improve > >>>>> performance given the amount of messages the brokers need to process > >> (in > >>>>> other words “throw more resources at it…”) > >>>>> > >>>>> > >>>>>> On 27 Mar 2023, at 18:28, Justin Bertram <jbert...@apache.org> > wrote: > >>>>>> > >>>>>>> What would you suggest is to do ? > >>>>>> > >>>>>> Did you try my previous suggestion already (i.e. using the > >> "group-name" > >>>>>> element in the "master" or "slave" element of "colocated")? > >>>>>> > >>>>>> Aside from that, do you know why you were asked to add another > broker? > >>>>>> Depending on the reason it may not be a good solution. > >>>>>> > >>>>>> > >>>>>> Justin > >>>>>> > >>>>>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <roy_co...@hotmail.com> > >>>>> wrote: > >>>>>> > >>>>>>> Hi Justin > >>>>>>> > >>>>>>> It is a good question I honestly don’t have the answer for. I > >>> inherited > >>>>>>> this configuration and was asked to add a third broker and to > ensure > >>>>> the co > >>>>>>> located backups are being done in such a way that each broker > points > >>> on > >>>>>>> another. Perhaps those who asked for it don’t fully understand > >> Artemis > >>>>> ! :) > >>>>>>> > >>>>>>> That said, those co located backup on the existing setup with two > >>>>> brokers > >>>>>>> do work as we have been enabled to recover lost messages in the > past. > >>> So > >>>>>>> even not optimal, technically it does work ? > >>>>>>> > >>>>>>> I can only imagine that those who initially designed it about 5 > years > >>>>> ago > >>>>>>> did not use a shared storage to avoid latency. > >>>>>>> > >>>>>>> What would you suggest is to do ? > >>>>>>> > >>>>>>> On 27 Mar 2023, at 18:00, Justin Bertram <jbert...@apache.org> > >> wrote: > >>>>>>> > >>>>>>> Your screenshot didn't come through the list. > >>>>>>> > >>>>>>> In any case, I'm pretty confused at this point. You're clearly > using > >> a > >>>>>>> colocated configuration that will request a backup from another > >> broker > >>>>> in > >>>>>>> the cluster, but you say you're not running multiple brokers in the > >>> same > >>>>>>> JVM. If you aren't running multiple brokers in the same JVM then > what > >>>>> are > >>>>>>> you using the colocated configuration for? The whole point of the > >>>>> colocated > >>>>>>> configuration is to run multiple brokers in the same JVM (i.e. a > >>> primary > >>>>>>> broker and also a backup broker for another primary in the > cluster). > >>>>>>> > >>>>>>> > >>>>>>> Justin > >>>>>>> > >>>>>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <roy_co...@hotmail.com> > >>>>> wrote: > >>>>>>> > >>>>>>>> I don’t believe we are. > >>>>>>>> > >>>>>>>> So assume three Virtual Machines on Azure. > >>>>>>>> > >>>>>>>> Each VM runs one Artemis broker > >>>>>>>> > >>>>>>>> > >>>>>>>> All of their ha policy section on all three brokers look like > that: > >>>>>>>> > >>>>>>>> <ha-policy> > >>>>>>>> <replication> > >>>>>>>> <colocated> > >>>>>>>> <max-backups>1</max-backups> > >>>>>>>> <request-backup>true</request-backup> > >>>>>>>> > >>>>>>>> > <backup-request-retry-interval>1000</backup-request-retry-interval> > >>>>>>>> <excludes> > >>>>>>>> <connector-ref>my-connector</connector-ref> > >>>>>>>> <connector-ref>thishostname.mydomain</connector-ref> > >>>>>>>> </excludes> > >>>>>>>> <master> > >>>>>>>> <check-for-live-server>true</check-for-live-server> > >>>>>>>> </master> > >>>>>>>> <slave> > >>>>>>>> <allow-failback>true</allow-failback> > >>>>>>>> <restart-backup>true</restart-backup> > >>>>>>>> <scale-down/> > >>>>>>>> </slave> > >>>>>>>> </colocated> > >>>>>>>> </replication> > >>>>>>>> </ha-policy> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jbert...@apache.org> > >>> wrote: > >>>>>>>> > >>>>>>>> We are not running multiple brokers on the same JVM but a single > >>>>> instance > >>>>>>>> > >>>>>>>> per VM, so each one has a dedicated JVM and VM > >>>>>>>> > >>>>>>>> Based on your previous message I was under the impression you were > >>>>> using > >>>>>>>> the "colocated" feature. *If* you're using this then you > definitely > >>> are > >>>>>>>> running multiple brokers in the same JVM because that's precisely > >>> what > >>>>>>> that > >>>>>>>> feature does. It runs a primary and a backup broker in the *same > >>> JVM*. > >>>>> If > >>>>>>>> you aren't using a "colocated" configuration then I'm not sure > what > >>> the > >>>>>>>> original question is about. Can you clarify? > >>>>>>>> > >>>>>>>> > >>>>>>>> Justin > >>>>>>>> > >>>>>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <roy_co...@hotmail.com > > > >>>>>>> wrote: > >>>>>>>> > >>>>>>>> Hi Justin > >>>>>>>> > >>>>>>>> Thank you for your input. > >>>>>>>> > >>>>>>>> Sorry, should have been clearer on our setup - We are not running > >>>>>>> multiple > >>>>>>>> brokers on the same JVM but a single instance per VM, so each one > >>> has a > >>>>>>>> dedicated JVM and VM > >>>>>>>> > >>>>>>>> Thanks > >>>>>>>> Roy > >>>>>>>> > >>>>>>>> > >>>>>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jbert...@apache.org> > >>> wrote: > >>>>>>>> > >>>>>>>> I'm not entirely sure if the configuration you want is possible. > You > >>>>>>>> > >>>>>>>> might > >>>>>>>> > >>>>>>>> try using the "group-name" element in the "master" or "slave" > >> element > >>>>> of > >>>>>>>> "colocated." Only servers with the same group-name will pair > >>> together. > >>>>>>>> > >>>>>>>> Aside from that I would actually recommend against using colocated > >>>>>>>> > >>>>>>>> brokers. > >>>>>>>> > >>>>>>>> The original use-case for this functionality was very early cloud > >>>>>>>> infrastructure where durable, attached storage was not readily > >>>>> available. > >>>>>>>> However, since then most (if not all) cloud environments support > >>>>> durable > >>>>>>>> storage separate from the broker so that if the broker goes down a > >>> new, > >>>>>>>> identical broker can be spun-up relatively quickly and attached to > >>> the > >>>>>>>> > >>>>>>>> same > >>>>>>>> > >>>>>>>> storage. This provides functional high availability without the > need > >>>>> for > >>>>>>>> any idle backups or replication of any kind which functionally > >>>>> nullifies > >>>>>>>> this feature. > >>>>>>>> > >>>>>>>> Additionally, it turns out that (surprise!) configuring & running > >>>>>>>> > >>>>>>>> multiple > >>>>>>>> > >>>>>>>> brokers in the same JVM is difficult and error-prone not to > mention > >>> the > >>>>>>>> complication of dynamically coordinating the acquisition of > backups > >>> in > >>>>> a > >>>>>>>> running cluster and protecting against split-brain. > >>>>>>>> > >>>>>>>> > >>>>>>>> Justin > >>>>>>>> > >>>>>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <roy_co...@hotmail.com> > >>>>> wrote: > >>>>>>>> > >>>>>>>> Hello everyone > >>>>>>>> > >>>>>>>> We have a setup of three Artemis brokers (very old version don’t > ask > >>>>> :)) > >>>>>>>> > >>>>>>>> We would like to configure the co located backups such that the > >>> backups > >>>>>>>> are sent in this order: > >>>>>>>> > >>>>>>>> Broker01 -> Broker02 > >>>>>>>> Broker02 -> Broker03 > >>>>>>>> Broker03 -> Broker01 > >>>>>>>> > >>>>>>>> > >>>>>>>> I was reading on co located backups here: > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>> > >> > https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html > >>>>>>>> > >>>>>>>> however not sure I fully understand how to configure the xml > section > >>> to > >>>>>>>> achieve that. > >>>>>>>> > >>>>>>>> Shall I add excludes in each broker, i.e. > >>>>>>>> > >>>>>>>> <colocated> > >>>>>>>> <excludes> > >>>>>>>> <connector-ref>...</connector-ref> > >>>>>>>> </excludes> > >>>>>>>> > >>>>>>>> Any help would be appreciated. > >>>>>>>> > >>>>>>>> Many thanks in advance ! > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>> > >>> > >> > >
