Hello Udo,
Thank you for your detailed reply and useful hints! The startup procedure will enforce the locators to start before the servers and the clients. Therefore, both servers and clients will have the complete locators list populated from the registry. I understand that this is not a standard/recommended way, but we have some technical constraints. To give you more details on the use case, we are using Geode as third-party library in a platform that already has its own middleware layer and its central registry. It’s not running in a virtualized environment, but we do not want to introduce additional deployment and configuration complexity for our clients. The registry monitors the status of the registered processes, and it is possible to detect locator failures. If a locator fails, it will be restarted on the same host and port. Therefore, there shouldn’t be a “new locator” issue. WAN replication is not a requirement in our case. But I didn’t check how the split brain would work in this case, I will give it a try and keep you posted. Best regards, Nadine 2016-06-22 21:18 GMT+02:00 Udo Kohlmeyer <[email protected]>: > Hi there Nadine, > This is an interesting scenario.... As you have already mentioned, the > locators property is recommended to be static and defined up front. > From what I understand the list of locators is build up dynamically at > runtime. i.e there must be some process that runs before the locators > property is set, that checks if there is an existing locator running and > populates its credentials into the property. > > If I "walk through" the start up process you have defined, it makes sense > that it could work. (so please don't take this as a blessing) It seems that > it would work for locators... but I'm not sure about what it means for the > servers or clients. > Also are you intending to us WAN replication? If so, how is that handled? > As the locators need a well-defined list of "remote-locators" at start up. > > It also brings up a whole bunch of other questions for servers, clients > and HA. > > How does the central registry handle locators failing/stopping? > > How does it handle the case if locator2 were to fail? I mean, locator3 > would still validly reference it, but it might not exist any more. (which > is fine). But if you now start locator4 (due to HA procedures), neither > locator1 or locator3 would know about locator4. (or locator5 or locator6) > > Does the central registry monitor the locators to confirm that they still > work and if not does it removes them from the list? What about connectivity > issues (split brain, etc...) > > Is the start up procedure for the clusters ALWAYS the same? i.e all > locators are started before the servers are started. Providing the "full" > list of locators to the servers at startup. > > How do servers handle the failure of locators? What happens if all > locators were to fail? The servers would still work, but reference > incorrect locator host names. So the servers would not know about any of > the "new" locators joining. > > How do clients get the full list of locators? > How are the pools defined? i.e do clients follow the same process? Check > the central registry for running locators and populate the pool locators > list with only the registered locators? > > What happens when clients start before all the locators have been started? > The ability for clients to connect/reconnect to clusters in case of failure > is through the defined list of locators/servers. If it detects failure in > the one, it will try the next in the list.... transparent failover. > > How does a running client handle the failure of all the locators? In case > of all locators failing, does it mean that all servers and clients have to > be shutdown and the whole eco-system has to be restarted? > > So maybe this approach is not advisable. > > If I may ask, why would you not be able to determine a set a servers that > are used for the cluster? Could you not somehow reserve some hosts for the > Geode cluster, without dynamic name allocation? Are you running in a > virtualized environment like AWS, VMware? > > --Udo > > > On 22/06/2016 10:24 PM, Nadine Daccache wrote: > > Hello everyone, > > > > I have a problem related to the locators property used to start the > members of a cluster, I thought you might help. > > > > In my case, the IP addresses of the locators hosts could not be statically > configured, and are only known at runtime. The idea is to register the > locator host and port in a central registry, and whenever a locator is > started, to check the previously started locators. > > > > For instance: > > - locator1 will be started with no locators property, and > registers host1[port1] > > - locator2 will be started with --locators=host1[port1], and > registers host2[port2] > > - locator3 will be started with > --locators=host1[port1],host2[port2], and registers host3[port3] > > > > I have tried it, and it seems to work fine. The 3 locators join the same > distributed system. But I haven’t performed more advanced testing. > > > > The documentation clearly states that every locator should have the > complete locators list, even if they have not been started. > > > > Do you know what would be the limitations (if any) if the locators are > started in this way? Do you have any idea that could help? > > > > Regards, > > Nadine > > >
