Hi Sijie,
Thanks a lot for the input - with your instructions I managed to get it working.
Pushing it a bit further using the standalone cluster with three deployments
starts failing with the message
17:03:21.400 [pulsar-io-50-1] ERROR
org.apache.pulsar.client.impl.ProducerImpl - [persistent://digital/dms/hello]
[pulsar.repl.standalone] Failed to create producer: Producer with name
'pulsar.repl.standalone' is already connected to topic
Given three clusters a, b and c all of which use “standalone” for their the
local cluster I can only send messages to one of the two others. When getting
the stats of the topic there is a replication backlog, I guess because of the
error above.
Fiddling around a bit I managed to change the default standalone cluster name,
please find the result at
https://github.com/chrsoo/apache-pulsar-standalone-geo-replication
<https://github.com/chrsoo/apache-pulsar-standalone-geo-replication>
I added kudos to you and to Brian as you both were instrumental to my
understanding - thanks for the help!
Cheers,
Christoffer
PS Tell me if you want your names removed...
> On 22 Nov 2019, at 17:17, Sijie Guo <[email protected]> wrote:
>
> Hi Christoffer,
>
> You don't need a global zookeeper to make geo-replication work. Thus, you
> don't need different cluster names as well since you don't need a global
> zookeeper.
> The existence of a global zookeeper is used for simplifying management of
> replication configuration.
>
> All you need to do is let your "standalone" clusters know how to connect to
> the other "standalone" cluster, by creating (registering) a cluster at each
> side.
>
> Here are the steps to configure geo-replication between two standalone
> clusters.
>
> 0) assume you have two standalone clusters, one is A, the other one is B.
> 1) at standalone A, create a cluster "standalone-b" using "bin/pulsar-admin
> clusters create". what it basically does is letting standalone A knows how to
> connect to standalone B.
> 2) at standalone B, create a cluster "standalone-a" using "bin/pulsar-admin
> clusters create". what it basically does is letting standalone B knows how to
> connect to standalone A.
> 3) at standalone A, create a tenant "my-tenant" and configure the allowed
> clusters for "my-tenant" to be "standalone" (standalone A cluster) and
> "standalone-b".
> 4) at standalone B, create a tenant "my-tenant" and configure the allowed
> clusters for "my-tenant" to be "standalone" (standalone B cluster) and
> "standalone-a".
> 5) at standalone A, create a namespace "my-tenant/my-ns" and configure the
> replicated clusters to be "standalone" and "standalone-b".
> 6) at standalone B, create a namespace "my-tenant/my-ns” and configure the
> replicated clusters to be "standalone" and "standalone-a".
> 7) util this point, you already setup a bi-direction replication between
> standalone-a and standalone-b.
>
> Using this approach, you are able to configure one-way replication or
> aggregated replication as well.
>
> - Sijie
>
>
>
> On Fri, Nov 22, 2019 at 7:36 AM Chris S <[email protected]
> <mailto:[email protected]>> wrote:
> Hi Brian,
>
> Thanks for the answer!
>
> Currently I don’t have any real error to show, mostly I am a bit confused and
> just need to know that I am not wasting my time trying to figure it all out.
>
> The cluster name is per default “standalone” for both my standalone clusters
> and I would think I need to setup different names for them for the
> geo-replication to work properly.
>
> So so I am trying to following the instructions on Pulsar geo-replication
> <https://pulsar.apache.org/docs/en/administration-geo/>, setting up tenants,
> namespaces and creating a new cluster with its own name etc as required. Then
> it seems I have to initialise the metadata for the cluster, with zookeeper
> URL’s and what not and it is somewhere here that I am starting to doubt that
> I am doing it the right way.
>
> Any additional pointers are appreciated!
>
> A few specific questions:
> Is the assumption that I need different names correct? Or can I use the
> “standalone” cluster?
> If I need to create a new cluster should I initialise metadata the same way
> as for a “real” cluster?
> In any case, thanks for the input so far!
>
> Cheers,
>
> Christoffer
> PS I also have issues making the pulsar dashboard to work properly. Basically
> I can see the standalone cluster but nothing else. No channels, tenants etc.
> To set it up I followed the kubernetes instructions
> <https://pulsar.apache.org/docs/v1.22.0-incubating/deployment/Kubernetes/>
> but adapted the broker configuration for standalone and removed all bookies,
> zookeeper etc. The broker, prometheus and grafana seems to work properly but
> not the dashboard.
>
>
>> On 22 Nov 2019, at 13:27, Brian Candler <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> On 22/11/2019 12:24, Chris S wrote:
>>> Does anybody know if it Is possible to do geo replication using standalone
>>> deployments in Kubernetes on a local workstation? Or are there limitations
>>> to the standalone deployment that prevents you from doing geo replication?
>> A standalone cluster is still a cluster.
>>
>> Have you tried turning on geo-replication between two standalone clusters?
>> If it failed, what error did you see?
>>
>
